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ABSTRACT 

Project REAL (Rural Education Aligned for Learning) aims to 
improve mathematics and science education in grades 5-8 in five rural Ohio 
school districts identified as in "academic emergency" or on "academic 
watch." Program goals are to reform curricula to teach skills and concepts, 
rather than facts; improve students' math and science competencies; increase 
high school graduation and college-going rates; increase teacher efficacy in 
teaching math and science; and enhance professional credentials of math and 
science teachers. This evaluation of the project's first year gathered data 
from two case-study schools, a survey of 136 teachers in all project schools, 
observations, interviews, and document inspection. Findings indicate that the 
project is encouraging the sorts of dispositions valued in current 
mathematics and science reform efforts. The project is engaging participants 
in all districts, and is influencing even teachers who do not teach 
mathematics or science. While some organizational cultures seem more 
auspicious for project success than others, the project is clearly helping 
many teachers to think systemically and is cultivating teacher-leaders. 
Finally, project staff are working hard to help districts improve teaching 
and learning while simultaneously working to improve the project's technical 
assistance capacities and responsiveness. Recommendations are offered for 
project sustainability. Nine appendices present a memorandum of 
understanding, planning document, surveys, and interview protocols. (Contains 
22 references) (TD) 
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EXECUTIVE SUMMARY 



Rural Education Aligned for Learning (Project REAL) conducted its first year in a planned 
five-year effort in 1999-2000, with support from the Ohio Department of Education. Project 
REAL is ongoing for 2000-2001; this report evaluates the Project’s first year of operation (1999- 
2000 ). 

Project REAL aims to improve teaching and learning in mathematics and science in 
grades five through eight in selected school districts located in rural southeastern (Appalachian) 
Ohio. Participating districts have been identified by the Ohio Department of Education as in 
“Academic Emergency” or on “Academic Watch” according to the rubric of the prevailing 
educational accountability system. All districts retained their initial 1999 classifications in 2000. 

The evaluation work focused on the five demonstration sites (districts) specified by the 
Project for 1999-2000. The major evaluation products are two case studies (described in detail in 
the full report) and an extensive cross-site teacher survey (with results detailed extensively in the 
full report). In addition, several visits were made to each of the five sites, the evaluator observed 
and participated in several monthly Project meetings, and he participated as an observer or 
participant-observer in Project conferences and training events. Methods used to gather data 
included classroom observations, participant observation, interviews, focus groups, a written 
survey about Project impact and depth, and document inspection. Initial visits were made in early 
October 1999, and the final observations and interviews were conducted in late August, 2000. 

Data analysis has been ongoing since the beginning of the evaluation. 

This report, in part, represents the voice of Project participants themselves, and the 
evaluator owes a long debt of gratitude to the frankness and generosity of many students, teachers. 
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principals, superintendents, community members, and Project staff and local Project team 
members with whom he interacted. The evaluation was designed as a so-called “participant- 
oriented evaluation,” which bases its judgements of worth or merit on the experience of Project 
participants themselves rather than on measurement of formal objectives' or on management 
needs. In fact, however, the evaluation is a “mixed model” since it also includes an extensive 
survey of Project impact and depth based on quantitative data. 

Participants’ Insights and Insights about Participants 

Participants (including teachers, students, administrators, and community members) tell 
rich stories about their connection to teaching and learning and about aspirations for themselves, 
their children, and their communities. The stories make clear that the Project enters complete 
worlds in which perspectives and commitments are already well established— for good and for ill 
alike. The Project aims to modify, adapt, or transform some of these perspectives and 
commitments (especially in the case of participating teafchers and administrators) in the eventual 
service of its stated goals. Participants, of course, also attempt to modify, adapt, or transform the 
Project’s perspectives and commitments for their own purposes. School reform is very much a 
negotiation, and the participants’ stories speak richly to that idea. 

Fortunately for the Project, however, recent teacher training and professional development 
practices, not to mention the messages from the state and national reform movements, reinforce 
the Project’s perspectives and commitments. Indeed, the Project is planned to augment those 



‘ Formal project objectives are listed in the Project’s “Request for Funding Assistance” (Ohio University 
Southern Campus, 1999). The objectives are planned for accomplishment by the end of the five-year 
Project period. In the first year of Project work, the Project director and evaluator agreed that a 
“participant-oriented evaluation model” was the most suitable. 
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efforts and is consistent with current professional thinking.^ 

Students, teachers, and administrators, however, inhabit worlds that are very far removed 
from both policymaking and pedagogical theorizing. Students, who spend a great deal of their 
lives inside school buildings, told stories that clearly indicated they want to be engaged, 
energized, and that they long to be inspired by instruction. Their stories also suggest that they 
regard such opportunities as so rare they hardly expect them. 

The Project is very clearly on students’ side in this respect. Most local Project team 
leaders (teacher-leaders and administrators) understand why students want this experience and are 
learning how to help give them more of it— not just personally, but organizationally.^ This is a 
new role for many teachers and they are clearly getting used to it. They find it simultaneously 
uncomfortable and rewarding. 

Some organizational cultures seem more auspicious for Project success than others. An 
organizational culture that is inherently less auspicious to the Project, however, seems not to 
disable the Project’s efforts. Instead, the efforts flow and accrue differently in that site (one of the 
five). 

A clear strength of the Project is that it is helping many teachers strengthen their ability to 
think systemically (that is, to consider the organizational implications of their thoughts and 
actions, and, thus, the thoughts and actions of others). Available evidence suggests that the 
Project is working to cultivate leadership, not simply relying on existing leadership to ensure its 
success. Good reform projects work this way; but too few projects succeed in this. Project REAL 



^ The Project has notably received substantial assistance from the mathematics, science, and technology 
leadership of the North Central Regional Educational Laboratory. 

^ The local Project team leaders reflect the Project director’s ardent intention in this respect, which 
constitutes evidence of the compatibility of the director and local leadership. This concordance also 
attests to the quality of communication on substantive matters that characterizes the Project. 
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seems already to be one of the exceptions. 

A few examples illustrate this success. A teacher interviewed at one site observed, 
“Project REAL is the most thrilling, engaging, and exciting, discovery- and project-based 
program I've experienced. What I appreciate most is that it treats the how and why of learning.” 
Another teacher, located at a site where organizational culture has tended to disable leadership, 
observed, “The Project has gotten us to work with each other more— we did this less last year. 
Lots of teachers are nervous but excited; the Project has provided evidence that something works. 
A lot of teachers are accepting the Project.” In another location, a teacher who was not on the 
leadership team talked about how her teaching has changed to incorporate more hands-on 
instruction precisely as a result of her understanding of what science is all about: “I teach this 
way because I’ve analyzed my understanding of science....! need to do that to see the big 
picture....! don't follow the textbook anymore, and I do many more labs.” The teacher is not 
strongly aware that some of the professional development opportunities that come her way are 
sponsored or amplified by the Project. The best change agents are often the least obtrusive. 

Themes and Issues in the Stories of Participants 

The evaluator induced five themes and three issues on the basis of interactions with 
participants. The “themes” are lower-order inferences that reflect the evident preoccupations of 
participants; the “issues” are higher-order inferences that encompass evident preoccupations but 
provide a framework for discussing (or, more particularly, for “problematizing”) dilemmas 
lurking in participants’ stories. Field notes were the chief source of data for deriving themes and 
issues. Findings from the cross-site teacher survey will be considered in a separate section of this 
summary. 

Themes . Themes prominent in the stories told by participants concern Ohio Proficiency 



testing and accountability, realizing constructivist pedagogy and curricula, engaging professional 
development, experiencing the ambiguities of change, and realizing collaboration and autonomy. 
So far as can be judged by the field notes, many of the Project participants interviewed are 
thoughtfully engaging practices that the Project promotes. This engagement is complex and 
challenging for many teachers largely because it is embedded in a set of contextual issues that 
influence how themes “play out” on the ground. 

Issues . Critical issues, as the evaluator infers them from the stories of Project participants, 
are the following three: organizational capacity, rural culture, and teachers' knowledge of 
educational change and reform. Whereas much of the Project work and activity is appropriately 
centered on classroom instruction, the Project is embedded in particular organizational and 
community contexts that are infrequently represented in the pedagogical thinking of classroom 
teachers. 

All the demonstration sites exhibit differing organizational cultures, and organizational 
cultures strongly influence the thoughts and actions of their members. Nonetheless, as the 
Project cultivates teacher-leaders (a process that is succeeding in all sites), their new roles 
inevitably lead them to ponder these issues. Teachers-leaders must continue to struggle with these 
issues as they try to improve achievement through more authentic curriculum and teaching. 
Administrators carmot lead schools towards reform alone. Indeed, administrative leadership is 
best constituted and powered by teacher-leaders. 

In addition, community engagement is a potentially powerful ally of reform (as one case 
study presented in the full report strongly suggests). Too often in rural school districts, we see the 
same degree of separation between school and community that characterizes schooling in other 
locales. The aforementioned case study, however, gives one example of how academic bridges 
can be built to link rural schools in responsive ways to their rural communities. As with engaging 



curriculum and instruction, authenticity is essential. There is no substitute for cultural knowledge 
and respect. 

Findings from the Cross-Site Teacher Survey 

The mid-year decision to focus qualitative work principally on developing two case 
studies permitted the evaluation to include a Project- wide survey intended to develop 
generalizable findings about the Project as a whole and about Project-relevant subgroups of 
teachers. All middle school teachers in all schools served by the Project completed an instrument 
negotiated by the Project director and the evaluator.'* In brief, the instrument collected 
information from all middle school teachers in the demonstration-site districts (whether or not 
they had teaching assignments in mathematics or science) about (a) Project-relevant 
demographics, (b) Project implementation and impact, and (c) organizational capacity related to 
district-level collaboration. 

Survey data indicate that Project REAL is strongly in evidence in the five demonstration 
site districts. Local Project REAL leadership team members were highly engaged in all Project 
sites (effect size = +2.0 in comparison to others), a finding that substantiates a high level of 
commitment among local teams. The next highest level of involvement was attained by math- 
science teachers (effect size = +1.0 in comparison to others). 

In addition, a substantial proportion of the variance in project impact^ (60-70% 
depending on analysis) was accounted for by three variables: (1) level of involvement in Project 

The instrument appears in Appendix C, and its development and administration is described in the full 
report. 

’ Project impact was measured as a composite variable consisting of the teacher’s current views of hands- 
on learning (as of May 2000); use of national and state materials to create activities and lessons (as of 
May 2000); change in thinking as a result of Project REAL (as of May 2000); and change in actual 
teaching as a result of Project REAL. Alpha reliability of this four-item measure was high. 



REAL, (2) change in view of hands-on-leaming in the 1999-2000 school year, and (3) having a 
math-science teaching assignment. Importantly, although math and science teachers are the focus 
of the Project's work, other teachers were also involved in Project activities to a measurable 
extent, some at very high levels. Findings strongly suggest that the Project is having impact 
schoolwide, beyond its focal domain in mathematics and science. 

The full report details differences in Project impact and participation among various 
subgroups of teachers. In particular, across all Project sites, age is shown not to exert a significant 
influence on impact. In addition, results of the survey show that Project impact was relatively 
high in four of the demonstration site districts. Other data suggest that a likely cause of lower 
impact might be some feature of organizational capacity unrelated to teachers’ collaborative 
practices. 

Recommendations 

This report provides five recommendations, derived from the stories of participants. The 
recommendations address these topics: sustain the level of commitment already begun, improve 
communications on three fronts, assist efforts to sponsor community engagement, gather baseline 
data for future evaluative purposes, and scaffold support for improved technology use at local 
option. The first of these is seen as essential in the eyes of Project participants, who value the 
record of commitment exhibited by the Project staff thus far. Teachers told the evaluator they 
had, in the past, seldom experienced this degree of “staying power” from a professional 
development project of any sort. Such remarks seem an authentic indicator of the Project’s value. 

Conclusion 



Interactions with participants strongly suggest that the Project is helping widely encourage 



the sorts of dispositions valued in current mathematics and science reform efforts. Data from the 
Project-wide survey of teachers show that the Project has actively engaged participants in all 
districts and, further, that Project impact is sufficiently strong to influence even teachers who do 
not teach mathematics or science. Finally, Project staff are working hard to help districts improve 
teaching and learning while simultaneously working to improve the Project’s technical assistance 
capacities and responsiveness. 
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PROJECT REAL FINAL EVALUATION REPORT 
1999-2000 SCHOOL YEAR 

Introduction 

Project REAL (Rural Education Aligned for Learning) is an effort to improve the 
instruction of mathematics and science in grades 5-8 among selected school districts in rural 
Southeast Ohio (Appalachian Ohio). The assumptions on which the program has been established 
and continues to operate derive from a constellation of psychological and sociological theory now 
known as “constructivism” (see, for example, Berger & Luckmann, 1966; Walkerdine, 1988). 

This theory has been enormously influential in national curriculum reform efforts, notably in 
mathematics and science, and is manifested in such documents as the National Council of 
Teachers of Mathematics's (NCTM) Curriculum and Evaluation Standards for School 
Mathematics (NCTM, 1989)] and the American Association for the Advancement of Science's 
(AAAS) Project 2061 Benchmarks for Science Literacy: A Tool for Curriculum Reform (AAAS, 
1995). 

In the application for funding to the Ohio Department of Education, Ohio University 
Southern Campus proposed the following explicit goals: 

1. reform curricula to teach skills and concepts, rather than facts 

2. improve students’ math and science competencies 

3. increase high school graduation rates 

4. increase college-going rates 

5. increase teachers’ efficacy in teaching math and science 

6. enhance professional credentials of math and science teachers. 

Objectives related to these goals are stated quantitatively, and begin to become measurable 
starting with the third year of the Project (Ohio University Southern Campus, 1999, pp. 4-5). The 
academic year 1999-2000 was the Project’s first year of operation. 



This evaluation report (1) describes the nature of the evaluation approach selected in 
concert with Project staff, (2) overviews the process of the evaluation, (3) presents case studies of 
two contrasting Project districts; (4) summarizes additional interviews and observations; (5) 
synthesizes emergent themes from the field work (site visits, interviews, focus groups, field notes 
from participant observation), (6) describes and analyzes results from the May 2000 cross-site 
teacher survey, (7) offers formative feedback for consideration of Project leadership, and (8) 
offers prioritized recommendations for consideration of Project leadership. The report appends 
documents relevant to the conduct of the evaluation. District and personal names are given 
pseudonyms in this report in order to protect the confidentiality of those with whom the evaluator 
spoke during the year. 



Evaluation Approach, Design, and Implementation 

Project REAL staff and the evaluator have agreed that a form of evaluation known as 
“responsive evaluation” (Stake, 1975) is most appropriate and useful for this first year of funding. 
The focus of such an evaluation is on project activities and the views of participants rather than on 
project goals, decision making for management, or the marketing or dissemination of any similar 
project in the future (cf Worthen et al., 1997). The tacit intent of such an evaluation is to 
provide a formative impression, from the perspective of participants in the project, to the Project 
staff and, via their interpretation of this report, the funding agency.® 

Other approaches (e.g., objectives-oriented or management-oriented approaches) are 



^Although the governing Memorandum of Understanding with the contracting agency makes clear that the 
only audience of this report is the Project staff, the evaluator is committed to taking steps to disguise the 
identity of both individuals and districts in these written materials. Pseudonyms are used in this report for 
all persons, all districts, and all schools. 
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deemed inappropriate for this project partly because the changes at which the Project aims require 
a minimum of several years to be reliably observed, and more than that for improvement to 
emerge that is hypothetically attributable to intentional organizational change. The application for 
funding reflects this thinking in its specification of goals and objectives. The evaluation is 
intended to provide a surrogate set of eyes and ears for the Project staff and, indirectly via 
synthesis of Project staff, for the funding agency during the activities of the Project’s first year of 
operation. This decision committed the evaluation to fieldwork and school visits as the principal 
modes by which data would be gathered. 

Field of focus 

For the 1999-2000 academic year. Project REAL is providing modest assistance to 15 
rural Southeastern Ohio districts. Substantially greater assistance is, however, provided to five 
school districts judged by the Ohio Department of Education as being in either “academic 
emergency” or on “academic watch” under the terms of the statewide educational accountability 
system now in force. This is one indication that the Project districts are comparatively low- 
performing, as one might expect them to be, located, as they are in rural southeast Ohio. The 
evaluation effort is focused exclusively on these five districts, knovm in the Project as 
“demonstration sites.” 

The strategic evaluation question therefore asks: 

What understandings of Project REAL emerge from the views of participants in the 

demonstration sites? 

The strategic question permits the evaluator to assess the value of the Project from the perspective 
of multiple, locally positioned stakeholders. In other words, the evaluation aims to develop a 
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“grassroots” perspective on the Project’s first year. 



Evaluation Rationale 

The evaluator realizes that instructional improvement is very difficult and that the sharpest 
barriers to improvement prevail in low-performing rural and urban schools and districts. 
Improvement entails not simply a re-tooling of methods and materials, but changes in teachers' 
understandings, beliefs, and, in many cases, long-time habits and dispositions (e.g., Kannapel, 
Coe, Aagaard, & Reeves, 1999). The kind of instruction desired by NCTM and AAAS for 
instance, is neither prevalent nor has its general superiority been demonstrated beyond doubt. It 
does represent the new thinking in the field, and it has substantially displaced the behaviorist 
approach that was previously common.’ 

Improvement is especially difficult in impoverished locales for a constellation of reasons 
centered what poverty simultaneously does to individual and community needs and the resources 
available to meet them. In short, the intensity of need escalates while the level of available 
resources plummets. In some part of the nation, the capacity to distribute school jobs is a source 
of political power (Duncan, 1999) because good jobs are so scarce. In many rural communities, 
the individualistic aspirations that support academic achievement are not strongly evident in the 
local culture (Theobald, 1997). In many rural places, expert assistance is not available locally 



’The evaluator is aware that the link between (a) improved instruction along constructivist lines and (b) 
improved school performance has not yet been adequately demonstrated. The evaluator, however, also 
shares the belief of Project staff that engaged and active learning is intellectually superior to an 
unwavering routine of recitation and lecture in a silent classroom. Between the extremes of a classroom 
organized exclusively for inquiry learning, on the one hand, and one organized to enforce silence and 
order (with no activity or student discourse), however, lie many possible combinations and instructional 
styles that have unknown influence on student engagement, the standardized test performance of 
individual students, and school or district performance in the aggregate. In general, the best opportunities 
for school and district performance on Proficiency Tests seem to lie in the improvement of the learning of 
children from impoverished families. 
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(Hadden, 2000). Some impediments can be overcome; but others seem require dubious changes 
to strong and functional rural cultures because they constitute the legitimate foundations of rural 
life-ways (Berry, 1990; Theobald «& Howley, 1998). Thus school improvement in impoverished 
rural communities faces unique challenges, some of which cannot be overcome but must be 
managed as ongoing dilemmas (Cuban, 1992). 

Widespread consensus nonetheless exists among educators (whether rural, urban, or 
suburban) that academic accomplishment is a valuable tool in life. This is not surprising, since 
academic accomplishment would seem to be a necessary focus of schooling. But many educators 
also believe passionately that children in impoverished communities would become more 
academically accomplished if they experienced a better sort of schooling. This is a more 
surprising view for a number of reasons. First, concern for the academic performance of 
impoverished children has a short history (almost everywhere in the world). Second, the 
American ethos often constructs poverty as the outcome of dissolute living. People are poor (on 
this hateful view) because they are lazy, or stupid, or both. They don’t deserve even the meager 
charity insufficient to maintain them (again, on this hateful but quite common view). Third, the 
fact that educators can ascribe academic failure to themselves (as those who operate schools from 
day to day) would be akin to lawyers widely acknowledging their role in the failures of the 
American legal system. 

This more uncommon belief— in the efficacy of better schooling to boost academic 
accomplishment in impoverished communities-is nonetheless common to Project staff and the 
evaluator. Its representation in the districts studied will become evident shortly. 

Evaluation activities 



The evaluator conducted a variety of activities, including (1) making school visits, (2) 



attending formal project activities, (3) developing survey instruments (4) administering surveys, 
(5) conducting interviews and focus groups, (6) writing field notes, (7) transcribing interviews, (8) 
identifying issues, (9) analyzing data, (10) writing and submitting the interim and final reports to 
the Project REAL director. 

Evaluation Design 

Responsive evaluations by their nature do not begin with a plan that specifies all data 
gathering activities in advance of the evaluation. Rather (as above) they commit to a certain 
disposition, best summarized by the evaluation question, by common understandings of client and 
evaluator, and the projected allocation of resources as given in the Memorandum of 
Understanding (see Appendix A). Evaluation objects (events, cases, documents, issues) emerge 
in interaction with participants, including Project staff. 

Following initial visits to all demonstration sites, the evaluator prepared a draft planning 
memorandum and solicited feedback from Project staff. This mid-evaluation planning document, 
finalized in mid-January 2000 (see Appendix B), established the parameters of the evaluation in 
light of participation by the evaluator in several project activities and interviews with the 
leadership of all demonstration site teams. 

These parameters further focused data gathering activities after January 2000 on three 
objects: (1) a cross-site (i.e., all five demonstration sites) teacher survey (see Appendix C), (2) a 
case study of one district's (Ambler Hill School District) innovative approach to math instruction 
in the 8th grade, and (3) a case study of a district (Ambler Valley School District) with low- 
performing district of comparatively high socioeconomic status, and (4) interviews with Project 
leadership at the various sites. 

During the course of the subsequent months, additional activities were negotiated and 



specified: (1) interviews with community members at Ambler Hill, (2) participant observation 
during the summer 2000 “Teacher Academies” sponsored by the Project, (3) interviews and 
participant observation at the three districts not selected for development as case studies, and (4) a 
concluding interview with the Project director. 

In-depth document analysis per se was not — in consonance with the chosen evaluation 
design (i.e., responsive evaluation) — a focus of the evaluation. Instead, the evaluator consulted 
documents to help inform analysis and interpretation of field notes and survey data. 
Documentation collected for this purpose has included the following: (1) mathematics program 
improvement reports (also known as “curriculum audits”) for each demonstration site district 
school involved in the project, for grades 5-8; (2) science program improvement reports for each 
participating school, grades 5-8; (3) Ohio Department of Education proficiency profiles (1997- 
1998, 1998-1999, and 1999-2000) for each demonstration site school and district; (4) Project 
REAL applications from each demonstration site district; (5) progress reports from three districts; 
and (6) miscellaneous artifacts from Project activities (e.g., agendas, memoranda, internal 
analyses and reports, district maps, newspaper reports of project activities, electronic mail 
interchanges, a down-loaded Education Week article shared with the evaluator by a team leader, 
and so forth). The nature of such artifacts naturally varies substantially from site to site. 

Site Visits and Event Participation 

All demonstration districts were visited at least twice. Case-study districts were visited at 
least five times each. Activities conducted during these visits will be described more fully in the 
case studies and in the consideration of emergent themes and issues surfaced by this evaluation. 

The evaluator participated in three monthly Project REAL meetings, one via a videotape 
prepared by Project REAL staff for this purpose and two in the field, one hosted by a 
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demonstration site district, and one hosted at an off-campus location adjacent to one of the 
demonstration sites. The evaluator also played the role of participant-observer in a two-day 
conference co-sponsored by Project REAL at Ohio University's Southern Campus, at which he 
served as a panelist. 

An initial round of visits to all demonstration sites focused on meeting with the team and 
district leadership (Project REAL team leaders, principals, superintendents) to explain the nature 
of the evaluation and establish entree to demonstration sites. The evaluator also discussed 
concerns of the local team leadership about the Project and about their concerns for, and the local 
context of, school improvement in math and science within their districts. These discussions were 
essential input to the decision to implement a cross-site teacher survey and to fashion the two case 
studies subsequently plaimed. 

The In addition to helping the evaluator observe dilemmas and challenges of Project 
operation, these events also provided ample opportunity for a variety of informal interaction with 
participants in demonstration sites and (to an unknown extent at the conference) with participants 
from the other 10 sites, and with Project consultants. Among the most instrumentally useful 
contacts were those with Gil Valdez, consulting on assessment from the North Central Regional 
Educational Laboratory. Dr. Valdez's analyses and on-site observations seem to accord with the 
sense of the evaluator and Project staff that the cases selected for study (the innovative approach 
to 8th grade math in one district and the anomalous test-score performance in another) would be 
salient to the evaluation effort. 

Subsequent to January, the evaluator has made a series of site visits to (1) observe the 
classrooms of most middle school science and math teachers in the case-study districts and (2) to 
conduct separate interviews with teachers. Teachers were selected for interview, in each case 
study, following observation in their classrooms. Teachers whom informants claimed were 



“traditional” were selected for interviewing, as well as teachers whom informants had claimed 
were “innovative.” Interviews followed a protocol developed for the purpose (see Appendix D). 
The actual course of interviews did not, nor were they intended to, rigidly follow the protocols, 
but instead followed a course of questioning suggested by the response to the first questions 
asked, which concerned knowledge of the Project (including facilitators of obstacles to 
improvement), evaluation of the Project so far, and the interviewee's reflection reflections about 
change along both personal and systemic dimensions. Each interviewee, in essence, has told a 
unique story at the instigation of common prompts. 

Time allocations for site visits will additionally permit about half the number of visits that 
have already been completed. The evaluator expects to visit all sites at least once more, with 
additional visits to the case study districts and probably to one other site. The evaluator also 
expects to devote 3 or 4 days to participation in the June Summer Teachers Academy. 



Communication 

The evaluator and Project staff also conferred regularly by telephone and email and met 
face-to-face several times to consider the ongoing evolution of the evaluation effort. Support staff 
were consistently helpful to the evaluator. 

Site visits were arranged personally by the evaluator with local team leadership. Team 
leaders were very generous with their time and arranged interviews with a wide range of people at 
the evaluator’s request. As a result, agendas for field visits were full and time was efficiently 
used. There were very few “surprises” upon arrival of the visitor, and the demonstration sites 
exhibited great flexibility in altering the agenda as these surprises dictated. 

The quality of communication with students, teachers, community members, and 
administrators seemed to the evaluator ideally suited to the purposes of this evaluation. 
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Interviews generally lasted up to 90 minutes, and seldom less than 60 minutes. Interviewees 
seemed very comfortable and their responses struck the evaluator as candid. Discussions 
exhibited laughter, puzzlement, irony, earnestness, and hope — the range of expression that would 
seem to warrant this view. The evaluator often left these interviews (and observations) with a 
feeling of reverence and deep appreciation for the difficult and thoughtful work that many 
teachers do. 

Data Collection 

Data collection, ongoing through mid-August 2000, received devoted attention throughout 
the evaluation effort. The evaluator prefers the method of taking notes as least obtrusive for 
interviewing and observation. The method requires that formal field notes be composed 
immediately after such encounters in order to assure authenticity and to preserve the advantage of 
memory, which is inevitably called upon to elaborate hastily made notes. Notes of this sort, in 
fact, serve principally to provoke the work of memory. With very few exceptions, field notes 
were composed the evening of site visits or the day following. 

The cross-site teacher survey conducted in May 2000 of course involved a quite different 
data collection strategy. Survey instruments were provided in sufficient quantity to conduct a 
census of all teachers at grades 5-8, plus a few teachers at other levels involved with Project 
activities. Project team leaders or principals provided teachers with the surveys. Completed 
surveys were returned anonymously, via the team leader, to the evaluator. All forms were 
returned by mid- June and analyses were conducted in September and October 2000. 

Response rate was over 90 percent, and the evaluator regards the results of the survey as 
fully representative of the teachers at Project schools. All teachers at Project grade levels were 
surveyed to determine possible ripple effects of Project REAL throughout the school. 
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Context 



Responsive evaluation attends closely to contextual issues. Initial site visits were 
extremely important for helping the evaluator develop a sense of the V2irying contexts in which the 
Project would unfold in the course of its first year. The contexts are indeed varied. Before 
reporting the evaluation’s interpretations, then, it seems wise to consider two sorts of context 
especially salient to the Project and its evaluation. 

The importance of context . Two senses of context are important. Since each context 
hypothetically introduces variability in implementation, the two contexts are usually related. 

First, the context of the Project's overall intention exerts an influence on implementation. 
Does the project in question seek to permit or restrict variability in implementation? 
Implementations can vary depending partly on what sort of variability in implementation a project 
might intentionally and unintentionally allow. Some projects aim for implementations that adhere 
as faithfully as possible to an idealized model. Project REAL is not among such projects; it is not 
aiming to replicate a proven “model” in similar fashion in all the demonstration sites, at least at 
one level.* In fact, the Project achieves a measure of responsiveness to the second sense of 



'Local demonstration-site teams are free to address a subset of Project-identified needs— or not to address 
Project-identified needs at all, in fact, should they wish. They might, for instance, have identified 
alternative needs using their own procedures (in fact, none of the demonstration sites have opted for this 
alternative, so far as the evaluator has been able to discover). They might have sponsored their own 
activities, or they might have participated vigorously in opportunities created by the Project (most districts 
have done some of both). There is a sense, however, in which fidelity has been achieved and promoted, 
this report may considers this issue as a critical sidebar on the problematic nature of educational change. 
The achieved fidelity, however, consists in this fact: all the demonstration site leaders and team members 
are taking very seriously the injunction to develop a more “hands-on” style of instruction. Many teachers- 
-even teachers outside the Project teams— are aware of these “new” methods, and many intuit that the 
change represents a sort of pedagogy somewhat at odds with their own classroom practice. This must be a 
disturbing realization for most teachers, even those most sympathetic to the recommended changes. Very 
few of those with whom the evaluator has spoken, however, can comment on the relationship of this 



context— locality— precisely because it places the burden of designing improvement activities on 
local demonstration site teams. 

The second sense of context, then, constitutes the varying organizational and community 
settings of the demonstration sites themselves. How do these differing contexts respond to the 
opportunity to address local needs in a fashion that is hypothetically more responsive to those 
needs than is usually the case with school improvement efforts? The evaluation literature is 
filled with the importance of this sort of contextual variability to quality or fidelity of 
implementation. That is, even where a project strives for high-fidelity implementation, one still 
expects some, even considerable, variability in implementation. 

Project REAL announces itself as an explicitly rural education project. Because the 
recent rural education literature (see Kannapel & DeYoung, 1999) has construed the cause of the 
contemporary “rural school problem” to be the success of 1960s- 1990s school reform efforts 
(which accorded little or no importance to context in general, let alone rural context), the 
evaluator is inclined to conclude that Project REAL has put the issue of its own intentions in 
proper perspective— put them “in context,” so to speak. The range of motion enabled or realized 
in actual implementations, however, cannot be so readily judged. 

Demonstration site contexts . Although all the demonstration sites have been classified as 
either on “academic emergency” or “academic watch”— which means they are, in essence. 



pedagogy to its theoretical base in cognitive psychology or— still more remotely— to its roots in the related 
epistemological and ontological perspectives known loosely as “constructivism.” This silence is not at all 
surprising in view of the conditions of practice and the intellectual habits of teachers and administrators, 
but it does mean that many teachers and administrators probably regard the injunction to develop a more 
“hands-on” style of instruction rather like an orphan infant who has appeared on the doorsteps of the 
orphanage. Why is it here? Where did it come from? They don't know, but they mostly believe that 
caring for it is a blessing. Interviews and survey results, as reported in this report, corroborate this 
interpretation. 
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academically low-performing districts in largely impoverished Appalachian Ohio-they are each 
quite different from one another. Table 1 provides information about this variability.^ 



Table 1 

Contextual Variability Among Demonstration Site Communities 

(Selected Measures) 



district 


test 1 


test 


2 mean 
income 


child 

poverty 


adults 

college 


sp. ed. 
students 


students 
per job 


Ambler Hill 


40% 


47% 


$22, 000 


40% 


5% 


16% 


9.1 


Upper Ridges 


42% 


53% 


$27,000 


25% 


3% 


11% 


10.4 


Lower Ridges 


43% 


47% 


$32, 000 


13% 


9% 


14% 


9.1 


Western Ridges 


37% 


45% 


$36, 000 


9% 


10% 


11% 


8.8 


Ambler Valley 


40% 


43% 


$53, 000 


2% 


20% 


9% 


10.8 



test 1: First achievement measure: the average of six district- 

level passing statistics, rounded to nearest integer (all grade 4, 

6, and 9 passing mathematics and science on the 1998 administration 
of the Ohio Proficiency Tests); i.e., the average of six scores, 
three in math and three in science (Source: calculated with 

information from Ohio Department of Education) 

test 2: Second achievement measure: the average of the percentage 

of students passing math and passing science on the first 1998 
administration (grade 8) only, rounded to nearest integer (Source: 
calculated with information from Ohio Department of Education) 

mean income: First measure of socioeconomic status: mean income of 

district households with children, rounded to nearest thousand 
(Source: 1997-98 Common Core of Data, U.S. Department of Education) 

poverty: Second measure of socioeconomic status: percentage of 

children in district in households living at or below the poverty 
line, rounded to nearest integer (Source: 1997-98 Common Core of 
Data, U.S. Department of Education) 

adults college : Third measure of socioeconomic status : percentage 

of the adult population aged 25 or older with educational attainment 
of at least a bachelor * s degree, rounded to nearest integer 
(Source: 1997-98 Common Core of Data, U.S. Department of Education) 

Sp Ed Students: Fourth measure of socioeconomic status: percentage 

of special education students, rounded to nearest integer (Source: 
1997-98 Common Core of Data, U.S. Department of Education) 



^ Identities of the five demonstration site districts are shielded through the use of pseudonyms. 
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Students per job: Measure of economic importance: Number of 

students per school district employee; computed from data in Table 2 
(column 6 divided by column 4). School employment is an important 
economic factor in rural communities (see Porter, 1997) . 



The baseline data in Table 1 show that quite dramatic differences— merely on superficial 
demographic characteristics— exist among these districts. While the overall academic 
performance is similar among the districts'®, this similarity is achieved under sharply varied 
contexts. 

Table 1 provides a contextually relevant view of the demographic feature (i.e., 
socioeconomic status) that has consistently been shown to exert the most influence on student 
achievement (see, for instance, Jencks, 1979). On all measures, for instance. Ambler Hill is by 
far the most impoverished of the five districts. Barely five percent of its residents have college 
degrees. In fact, a large percentage— perhaps most— of the district's college-educated residents can 
be found among the teachers and administrators of the district. Despite this fact. Ambler Hill is 
not the lowest performing district on the global achievement measures reported in Table 1 . 

Ambler Hill School District is currently classified by the Ohio Department of Education as 
under “academic watch.” The district's innovation for eighth grade mathematics, “Math Lab,” is 
the focus of one of the evaluation's two case studies. 

Ambler Valley is a contrasting case. It is the most affluent of the five districts, again by 
every measure. Average household income among families with children is $52,000, 236% 
higher than that in Ambler Hill. Fully .,4^% of adults residing in the district have college degrees, 
and the child-poverty rate is quite low, actually at less than 2 percent. Ambler Valley does not 
demonstrate the highest achievement measure, but its “8 only” measure is the lowest among the 



In particular, the contextual differences between Ambler Hill and Ambler Valley districts are large. But 
their 1998 performance in mathematics and science is similar. 
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five districts. In fact, on the basis of prediction from socioeconomic status, Ambler Hill might be 
said to perform beyond expectancy and Ambler Valley below expectancy. Indeed, on the same 
basis. Upper Ridges School District could also be said to have been doing quite well in 1998. 

The remaining two districts. Lower Ridges and Western Ridges, were not so impoverished 
as Ambler Hill nor so affluent as Ambler Valley, and their global math and science achievement 
levels were also in the middle range. Western Ridges, it might be noted, exhibited the lowest 
1998 “4-8” achievement measure, however. Western Ridges School District is the only county- 
wide district among the five demonstration sites. Again, however, the variation in achievement 
scores viewed this way is not nearly so great as variation in demographic measures. 

Table 1 also includes a measure of the salience of employment by the district to the local 
economy, “students per job.”" One might interpret this statistic as the economic efficiency with 
which students are used locally to generate local jobs. All else equal, the lower the ratio on this 
measure, the more employment opportimity generated by each student. The ratio for these five 
districts varies from 10.8 students per job (Ambler Valley) to 8.8 students per job (Western 
Ridges). This means that 100 students in Ambler Valley, on average would generate 9.3 local 
jobs, whereas 100 students in Western Ridges would generate 1 1 .4 jobs, or two additional jobs 
with decent rural salaries in Western Ridges as compared to Ambler Valley. 

The districts with the two highest ratios (i.e., where students are a less efficient source of 
local jobs) are, in fact, within easier commutes of better jobs (i.e.. Upper Ridges and Ambler 
Valley, each located outside sizable towns with retail and manufacturing employment). On the 
other hand, it is the largest district— the county district of Western Ridges— where students serve as 



" A commonplace observation in the research literature on rural schools is that schools are often the 
largest local employer. The economic importance of students as a source of local jobs is often overlooked 
by the mainstream literature on schooling, but when this circumstance is not overlooked, it is usually 
viewed as an impediment to school improvement (e.g., Duncan, 1999). 
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the most efficient source of local jobs among these five districts. 

Table 2 provides a closer-in view of the structural features of the districts themselves. 
This information is useful to the evaluation, since district size has been shown to influence 
Proficiency Test performance in Ohio in a study sponsored by the Rural School and Community 
Trust (Howley, 1999a, 1999b). 



Table 2 

Organizational Variability Among Demonstration Sites 
(Selected Measures®) 



district schools 


locale 


students 
(number, rank) 


S/T 

ratio 


total 

employ 


SS/T 

ratio 


Ambler Hill 


2 


{ rural )^ 


1,000 


(500) 


20:1 


110 


1.1 


Upper Ridges 


3 


rural 


2,000 


(290) 


18:1 


192 


0.7 


Lower Ridges 


2 


rural 


1,000 


(510) 


19:1 


110 


1.2 


Western Ridges'" 


10 


rural 


5,500 


(60) 


16:1 


627 


0.8 


Ambler Valley 


3 


small town 


1,500 


(370) 


22:1 


139 


1.0 


schools : 


number 


of schools 


operated 


by district 







locale: type of locale assigned by the National Center for Education 
Statistics 

students: number of students enrolled by districts, rounded to 
nearest hundred; Ohio district size rank in parentheses (l=largest; 
611=smallest ) 

S/T ratio: ratio of number of students to number of classroom 
teachers, rounded to nearest integer 

total employ: number of persons employed by district, full-time 
equivalency rounded to nearest integer 

SS/T ratio: ratio of other staff to classroom teachers, rounded to 
nearest tenth (i.e., other staff as percentage of teaching staff) 



Notes : 

® All data from the 1997-1998 Common Core of Data, U.S. Department of 
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Education . 



Ambler Hill School District is classified by NCES as being located in 
the "urban fringe of mid-size city" by NCES. AHSD is nonetheless indeed 
very rural, with no light industry, no fast-food restaurants (indeed, no 
restaurants at all), many small farms, few villages, no traffic lights, 
and no villages with populations over 1,000. The classification is an 
artifact of the district's location in a county that is adjacent to a 
county located in a census-designated metropolitan county. An declining 
industrial town is about 25 minutes from the school and some residents 
take that daily commute. AHSD is more remote from such a center than 
either Western Ridges or Ambler Valley. 

Western Ridges School District is a county-wide district. 



Table 2 shows the variability in organizational structure that exists among the 
demonstration site districts. Size variability may not seem great, if one looks only at the numbers 
of students enrolled. However, the data (all from the same source and year as the demographic 
data in Table 1) also include the rank of district size among Ohio's 611 districts (given in 
parentheses in column 4 of Table 2). 

The rank data show that these five districts include one district that is, in fact, near the 
90th percentile of district size in Ohio (i.e.. Western Ridges School District), two that are near the 
average (i.e., 50th percentile) district size in Ohio (i.e.. Upper Ridges and Ambler Valley school 
districts) and two that are between the 10th and the 20th percentile of district size in Ohio (i.e.. 
Ambler Hill and Lower Ridges school districts). Table 2 shows, as well, that the size of the large 
district is an artifact of district type— Western Ridges is a rural district that covers an entire county, 
a fact that is reflected in the larger number of schools operated there (i.e., 10 as compared to the 2 
or 3 schools operated by the other districts). 

Student-teacher ratio (column 5 of Table 2) also shows modest variability, varying from 
about 16 students per classroom teacher to 22— which indicates that each teacher in Ambler Valley 
is, on average, responsible for 37.5% more students than each teacher in the Western Ridges 
School District, and about 10% than in Ambler Hill. 
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Support-staff-to-teacher ratios,'^ reported in column 7 of Table 2, reflect several 
conditions simultaneously: (a) concentration of adults per student; (b) allocation of human 
services, and (c) both administrative and blue-collar concentration. The ratio does not separately 
define these influences, but rather indicates a cross section (variously constituted) of adults also 
available locally to (as the definition given by the U.S. Department of Education puts it) “nurture 
students.” The extent to which such additional nurture might actually take place, of course, 
depends largely on organizational culture and climate. Interestingly, the districts with the lowest 
student teacher ratios (i.e.. Upper Ridges and Western Ridges districts) also have a lower ratio of 
support staff to teaching staff. 

Most interestingly for the purposes of the case studies. Ambler Valley exhibits the highest 
student-teacher ratio (i.e. larger class sizes) and a comparatively lower support staff-to-teacher 
ratio (i.e., 22:1 and 0.8:1, respectively), two facts that might together indicate that lower 
opportunities for nurturing students prevail in this district. Greater organizational effectiveness, 
of course, could be sufficient to counteract this inherent disadvantage. Ambler Hill, by contrast, 
has a comparatively higher student-teacher ratio (larger class sizes, second only to Ambler Valley) 
but also a higher support-staff-to-teacher ratio. 

The one structural characteristic that exhibits little variation is the rural and small-town 
character of these districts. All are essentially rural districts, though the rural character of the 
Ambler Valley School District has been rapidly changing over the course of the previous 10 or 15 
years. In the 1970s, interviewees reported, students were largely the children of farmers or factory 



“Support staff’ includes all central office personnel (e.g., superintendent, curriculum staff, and other 
administrators), principals, guidance and library staff in schools, as well as unclassified staff. 

Student-teacher ratios can be understood as a proxy for class size; lower ratios indicate lower class size. 
Lower support-staff-to-teacher ratios, on the other hand might (under some circumstances) indicate the 
availability of less support from outside the classroom for student nurture. 
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workers. Today, few farms remain and many more students are children of professional workers. 
This district is, in fact, located at the edge of the Appalachian region. The small town that is its 
center is thriving by comparison with many small rural towns--few homes, for instance, are on the 
market and practically all of them are superbly maintained. Surrounding the town, the former 
farms are rapidly becoming sites for houses that are generally less modest than those in town. 
Strictly speaking. Ambler Valley is neither quite properly rural nor quite properly suburban. In 
the evaluator's estimation it appears more rural than suburban, but appearances are deceptive and 
the district is becoming increasingly more suburban in character. 

Elsewhere among the demonstration sites, there is no evidence at all of this sort of ex- 
urban transformation. All the other districts comprise very rural areas with either distinctly 
depressed small-towns or stable small-towns and rural populations. 

One more contextual observation is necessary. Each of these school districts operates new 
schools, or (like Ambler Valley) is preparing to occupy new schools shortly. Some of these new 
facilities are set in what many people might regard as stunningly bucolic settings. The newer 
buildings appear to be faultlessly maintained; the Ambler Hill building, for instance, though now 
in its seventh year of occupancy, looks and feels like it opened this year. Though other rural 
districts in Appalachian Ohio are in dire need of major maintenance efforts, reconstruction, or 
new construction, all of these districts seem to have confronted the issue of deteriorating facilities 
at least to some extent, within the past decade— many quite recently. 

Case Study Reports 

This section of the report aims principally to tell the relevant stories of the people with 
whom the evaluator spoke in the two case-study districts. Such work occupies the center of a 



responsive evaluation and the goal here is simply to provide a voice (relevant to the foci of the 
case studies) for participants at those sites. The discussion begins with the Ambler Hill School 
District. 



First Case Study: “Math Lab” in the Ambler Hill School District 

Ambler Hill School District (AHSD) gives the classic rural impression that life here is 
idyllic, at least as seen from the northern approach to the district. Generally, this impression of an 
“idyll” is wildly inaccurate, since the association of agricultural production and poverty is strong 
throughout the nation. The demographic data reported earlier, in fact show the AHSD as the 
poorest of the five demonstration sites by a wide margin. The district leadership also assures the 
visitor that there are pockets of very sharp poverty throughout the district.'^ Apparently, the 
national association holds up locally in this case. 

Many families here continue the operation of small farms (cattle, tobacco), though income 
from this occupation is meager. Depressed factory towns, however, are within a half-hour drive, 
and many families appear to have some employment there. There is no industry and no 
professional employment in the district, according to the Project team leaders, Bethann Evermyer 
and Jamie Jackson, who teaches the focus of this case study, the “Math Lab” arrangement. 



‘‘‘ The major section of the report titled “Emergent Themes and Issues ” presents the evaluator’s overall 
interpretations of the based exclusively on the evaluation field notes. Findings of the cross-site teacher 
survey are considered in a separate section of the report. 

” The bottomland near the school, and lining the northern approach to the campus, really is picturesque, 
if hardly affluent. Fields are worked, and most houses are well maintained. During one visit, the 
evaluator left the campus for a 90-minute driving tour of the district. In the northeast half of the district 
ridge-top lands prevail, and the drive shows much poorer neighborhoods and properties here, with many 
operating oil wells in evidence. Superficially, less care and pride seem in play here than in the 
bottomland farms. Such appearances, however, can be quite deceptive, in the evaluator’s experience. 
Care and pride can take many forms. 
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Jamie appears to be a team leader of some sort, but is not, at an initial meeting with the 
evaluator, actually identified as such in so many words, either by Bethann or Jamie herself Her 
demeanor, however, is confident and forthright. Bethann, who now works in the district office 
(attached to the high school just across the parking lot), insists, “This is the teachers' project, not 
mine; nobody has ever given them $50,000 to spend, and it's interesting to watch them decide 
what to do with it.” Bethann served formerly as the principal of the K-8 building. 

The field notes observe, 

BE [gives the impression that she is] keeping on eye on things for a number of reasons: it 
was inherently interesting to watch the professional dynamics; these were her friends— \ 
think she said that she and JJ were “best friends”; and BE said sensible down-to-earth 
things about schooling and community. For instance: 

“Good teachers help kids learn; they could probably teach without any materials', we can't 
get enough good teachers, and we can't predict who will become a good teacher very 
well.” 

“We're all into each other's business here. What goes on in the school is a major topic of 
interest in the community. It's good and it's bad, and it's who we are. Mostly, people 
really like each other or really hate each other, but we all know nearly everything about 
each other.” 

The initial meeting at Ambler Hill seemed remarkably open.'® Betharm made remarks 
that one encounters rarely, and which may indicate a level of uncorrunon comfort or security in 
role. She said, for instance, 

I know we're working awful hard, but I sometimes wonder if we're not just 
spirming our wheels and if things will ever change much.” 

And, as the evaluator was taking his leave, she remarked. 

I'm not really a 'people-person'; I'm not sure I really like people all that much. 



'® Bethann Evermyer is a former student of the evaluator's spouse, and the evaluator is uncertain if this 
prior history might have influenced the quality of the initial meeting. Classroom observations and teacher 
interviewees, to be reported shortly, indicate that this connection is probably not influencing the 
evaluator's perceptions or the reports of teachers interviewed. 
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Sometimes I think I'd just like to sit up on my hill behind my fence and not see 

anybody. 

The latter remark is particularly interesting in light of the principal's self-description in an 
interview later in the year. Bill Fletcher was hired out of retirement the week of the evaluator's 
first visit (early October 1999). Bill said that he'd “thought he was ready for retirement,” but 
discovered after seven months that he was not. A former high school principal in one of the old 
industrial towns where district residents find limited employment, he said, “I'm a 'people-person' 
and retirement was driving me nuts!” 

The division of labor between Betharm and Bill is not precisely clear to the evaluator,'’ but 
it seems that it may rest on partly on this difference in avowed personality. Betharm, incidentally, 
is affable and friendly to both teachers and students, greeting students whenever walking the halls 
accompanied by the evaluator. That is, her self-image as “not a people person” was not evident to 
the evaluator in her work-time behavior. 

When it comes to affecting teachers beyond those who are members of the Project team, 
both Jamie and Betharm agree that the district employs perhaps six teachers who are not amenable 
to any sort of change. Without these teachers, both interviewees thought, other teachers would be 
more apt to follow the lead of natural teacher-leaders. 

Staffing is reportedly very stable in Ambler Hill, despite the fact that the district offers the 
lowest-paid teaching positions in the county. According to Betharm, just one teacher left Ambler 
Hill the past year, to a neighboring district in which that teacher lived. For some reason, low 



” Bill seems to manage the elementary school, whereas Bethann, the former assistant principal, is 
concerned district-wide with curriculum and instructional improvement, though she seems to spend much 
of her time at the elementary school. 
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salaries do not seem to be an incentive for staff to seek employment elsewhere.'* 

On the other hand, the district reportedly offers the highest wages in the county to non- 
professional staff. (According to the data in Table 1 and Table 2, it also offers proportionately 
more of these higher-paying jobs to local people than some of the other, more affluent districts). 

If these reports are correct, the result is a wage scale that would appear to diminish the status 
differences of professionals and staff; which, in other words, would hypothetically produce an 
egalitarian effect in the organization. In addition, proportionately more of the adult population is 
involved, as employees, in sustaining the school and the district. School Board members are 
“blue-collar” individuals who reportedly are suspicious of the professional expertise of teachers. 
They seldom visit schools or classrooms, according to Jamie and Bethann. Not all board 
members have educational attainment of 12 years or higher, apparently. 

Reception of the Project in Ambler Hill . So far as the evaluator could judge from 
speaking with Bethann Evermyer and Jamie Jackson, the Ambler Hill team was eager to 
participate in the Project because it coincided with an ethos of teaching and learning that made 
sense at least to the two team leaders. The evaluator, in fact, learned of the existence of the “Math 
Lab” arrangement during his initial visit. He was astonished to learn that eighth grade students— 
all eighth grade students— received two hours per day of mathematics instruction. 

Probing this arrangement, he discovered that the second hour was devoted to mathematical 
applications and to projects in which mathematics was the embedded focus. Data collection, 
problem-solving, connections among mathematical procedures and ideas— these seemed to be the 
work of “Math Lab.” The names of a number of teachers trying similar or related applications in 



Decent salaries, in Herzberg's theory of motivation (Herzberg, 1 966), are known as “hygienes” because 
they do not create or substitute for satisfaction in the work, but tend instead to minimize dissatisfaction. 
In this light, the absence of staff defections to other districts, would be prima facie evidence that 
organizational satisfaction is comparatively high in the AHSD. 



math and science were mentioned. By the time of the evaluator's first visit, in fact, one of the 
consultants who had participated in the Project REAL summer conference had been hired by the 
AHSD to provide assistance with a mathematics curriculum-mapping project. 

Jamie Jackson posed a number of critical questions about the Project to the evaluator: 

• Are we as 'far along' as other districts? 

• Can we get more guidance in completing the Project? 

• What documentation will be required? 

• What would NCREL's role be? 

• Would the Project fimd training (that was what they really needed)? 

• Would training be delivered at the school? 

While the evaluator was hardly able to answer these questions definitively, he made some 
suggestions about how to get answers and some predictions (e.g., about training and the NCREL 
role). More to the point, however, is the observation that questions so clearly framed and frankly 
stated indicate a healthy level of engagement with the Project, in the evaluator's judgement. 

Decision-making in the Ambler Hill implementation of the Project . Bethann Evermyer's 
claim that the Ambler Hill implementation rested in the teachers' hands seemed to be home out by 
inferences from the evidence gathered in subsequent site visits, classroom observations, and 
interviews. The evaluator has not attended any formal meetings of the Ambler Hill Project team, 
but the organizational culture revealed in visits, informal conversations, classroom observations, 
interviews and in the focus group of 8th grade students manifests the existence of a substantial 
collaborative spirit. 

Teachers interviewed, for instance, talked not just about their ovm classroom practice, but 
about the classroom practice of colleagues— closely allied colleagues and other colleagues— in 
terms that are simultaneously respectful and critical. Everyone with whom the evaluator spoke 



(including both members and non-members of the Project team, and younger, mid-career, and 
end-of-career teachers) indicated a degree of engagement with the practice and also the ideas of 
hands-on learning. 

No interviewee claimed that students “couldn't handle” very much instruction of the sort 
valued by the Project staff— a an observation voiced by a teacher in another district. In addition, 
the excuse given for failing to “do more hands-on lessons” was always personal lack of capacity, 
experience, or knowledge of hands-on methods. Lack of materials was not frequently given as a 
reason. In fact the two teachers who did mention materials at all are among those most engaged 
in doing hands-on lessons. 

Finally, none of the teacher interviewees— not one, whether Project team member or non- 
member— claimed that they felt uninformed about the purposes, intentions, or progress of the 
Project. In fact, the AHSD had submitted the names of 30 teachers who wanted to participate in 
the Project REAL Siunmer Academy. 

The faculty at Ambler Hill, the evaluator infers, is well-connected to the purposes and 
intentions of the project. This statement hardly applies to every teacher, nor does it apply to the 
same degree to everyone, but, in the evaluator’s judgement, the general level of engagement 
seems high across the organization. 

Of course, engagement might be high, and participation low, but additional evidence 
indicates otherwise. The teachers' union seems to have a strong voice and strong leadership, in 
fact, from teachers who are quite capable in the classroom. The union has adopted a co- 
presidency for its leadership, the Math Lab teacher, Jamie Jackson and another middle-level 
teacher, Melissa Lenville. Another teacher, in whose classroom the evaluator observed, is vice- 
president. The middle-level teachers lead the union effort for the AHSD, in other words. It is not 
difficult to believe that the informal networks of teachers augment their strong formal 
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organization and that together, formal and informal networks exert a strong influence on 
organizational culture. 

The “Math” Lab phenomenon . Jamie Jackson teaches three periods of “Math Lab” each 
day. Bethann Evermyer attributes the district's climb from “academic emergency” to “academic 
watch” to the Math Lab. 

When eighth grade students took the ninth-grade Proficiency Test last year in 1999, they 
attained a 91% pass rate, an increase of about 40%. This is a robust difference and Greg Morton, 
the superintendent, confided that he was hopeful that it was not a “flash in the pan” and 
simultaneously fearful that it was. High hopes, then, ride on the results of the Proficiency Test 
administration for 2000, just concluded.'^ 

The district abolished study hall for eighth-grade students and also eliminated what they 
had called “intervention classes”- remediation classes for the Proficiency Test, presumably-and 
enrolled every eighth grade student in two math periods, a kind of mutant block-scheduling for 
this level. In effect, the decision was made to provide a richer mathematics experience for 
everyone instead of providing remediation for a few. 

The first period of mathematics instruction consists more or less traditional classroom 
activities. The second period involves math applications. Jamie seems to have a good sense of 
what this term means. Surprisingly, she claimed not to have been inspired by anything except 
intuition and her own judgement of what kids needed to learn math better: 

They collect data and apply math to “real-life” situations. The grade 7-8 math 

curriculum is “integrated math.” [The evaluator asks if it includes algebra] Yes 



’’ Ironically, the district learned during the course of the evaluator's involvement (since October 1999), 
that these 1 999 scores will not be reported until the 2001 report card. The reasoning appears to be that 
the 1999 8th graders are not really ninth graders in the year of the first administration of the (soon-to-be- 
discontinued) ninth grade test. They are ninth-grade students this year (2000), and so their 1999 
performance will be reported along with the results of the 2000 administration to other 9th grade students, 
that is, in the late winter of calendar year 2001 . 
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[Jamie replies] they get algebra through solving linear equations--no quadratics— 

but we don't call it “Algebra.” 

Four Math Lab lessons were observed on two days several weeks apart —both unannounced 
in advance. The administration of the ninth-grade Proficiency Test (and others as well, in fact) 
took place between the two days of observation. 

First Math Lab observation . The first observation occurred just prior to the administration 
of the 2000 ninth-grade Proficiency Test (being taken for the first time by these students the 
following week). The content of the regular (non-lab math) class was not observed, but the Math 
Lab on the day of the observation was— ironically— devoted to review for the Proficiency Test. 
Jamie Jackson was apologetic, and offered the opinion that the evaluator might want to observe a 
more typical Math Lab class on some other day. Partly at issue, however, was Jamie's overall 
approach to teaching. 

This supposedly “traditional” class, in the event, was very rapid-paced, filled with 
discourse and ideas, and almost totally engaged by the students, who seemed to the evaluator 
energized by the pace and quite devoted to their teacher. The substance of the class consisted of 
perhaps 1 5 “word problems” of various sorts, in multiple-choice format, which the students were 
to work quickly. Jamie distributed her questions widely among the group and elicited from them 
their problem-solving logic, test-taking strategy (sometimes, though more often she indicated 
possibilities), and answer— sometimes in that (reversed) order. At the very end, Jamie asked the 
students to calculate their own scores, observing that 75% right is the minimum goal. None who 
report their answers have gotten fewer than 83% correct. 

If this was a “traditional” lesson, the evaluator would not actually have recognized it as 
such. There was high academic focus, student engagement, rich discourse, and maximally 
product pacing. The class was energetic. In the evaluator's experience of school, a “traditional” 



approach to this lesson would consist in giving a test to students, scoring, and only then debriefing 
it (if at all). A traditional approach of that (usual) sort, one might observe, is more likely to be 
counterproductive and may, in fact, have non-academic purposes related to hidden curriculum. 
Jamie Jackson's approach was quite different from the usual one. 

In fact, in Jamie's version of Proficiency Test review, the “constructivist” ethos of helping 
students grasp— and make— mathematical meaning was well in evidence, despite the total absence 
of “hands-on” learning. In other words, Jamie Jackson would appear to be an unusually strong 
middle-school mathematics teacher, one who demonstrated, at least to the evaluator on this 
occasion, a high level of concern for the learning— the achievement-of all students. Nothing 
about the performance seemed feigned or simulated; it appeared in every way to be a typical 
instructional sequence for this teacher. Students were oblivious to the presence of the evaluator 
because the instruction was so engaging. 

Second observation of the Math Lab . The second observation occurred mid-way through 
a three-week project. The project consisted of teams of students fashioning stained-glass panels. 

It occurred after the Proficiency Test administration. In a sense. Math Lab had become an art 
class— and in a medium not usually represented in public schools (at least not in the evaluator's 
experience). Students had learned to design panels, cut and shape glass, and to solder the pieces 
together. The mathematics connection focused on the preparation of a poster to describe the 
geometric properties of the design. Students were just as engaged as they had been in the 
previously observed class, but were of course, up and about, collaborating, conferring, carrying 
out tasks. The room was noisy as a result, but the buzz was focused on the work at hand. 

The evaluator asked one of the two classes observed this day, “How many of you knew 
how to solder before this project?” Two students raised their hands. “So,” he continued, “the rest 
of you learned to solder, yeah?” Heads nodded. One student volunteered, “Ms. Jackson can teach 
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anyone anything?' 

This was a very “hands-on” class with a high level of student engagement. The evaluator 
personally wondered if the prepared poster of mathematical characteristics of students' stained- 
glass panels was sufficient justification for spending three weeks on the project. The 
mathematical content seems slight, for instance, in comparison to the more traditional class. 

The evaluator's answer to his own question was: “Yes.” It is true that the mathematical 
content is comparatively slight. At the same time, however, project-based learning is inherently 
less efficient^® than direct instruction. Moreover, this interlude simultaneously demonstrates to 
students several memorable lessons of which they are probably unaware. Jamie herself could be 
unaware of them. 

First, the project shows students the relationship between competence in a school subject 
(mathematics) to their own competent teacher's relationship to life. That is, their teacher's skill in 
mathematical knowledge enriches her own life and the students get to glimpse that elusive quality 
in action. Nobody makes a “big deal” of this delicate experience, yet it happens. One might 
consider this embedded lesson a social studies lesson, and not a math or art lesson. 

Second, this project links-in practice~a realm of aesthetics (stained glass art) to a realm 
mathematics (geometry, in this case, but hopefully to mathematical ideas more generally as well). 
In his own schooling, the evaluator cannot recall a mathematics teacher ever successfully making 
this connection, and yet the connection is very important in the minds of practicing 
mathematicians (e.g.. Hardy, 1969). The connection between mathematics and beauty is an 
abundantly subtle point, and one that observers would scarcely believe possible in the 
mathematics instruction of a small, comparatively isolated, rural school. 

Third, the project puts mathematical competence side-by-side with practical competence. 

Meaning less “coverage” per time unit. 



and the notion that students would learn to solder in math class strikes the evaluator as most 
appropriate. In impoverished communities—like Ambler Hill— academics are far too often 
construed as antithetical to local ways of being. Putting a practical skill like soldering into math 
class, while it may seem arbitrary, is in fact more respectful than much else that occurs in many 
math classes. 

A teacher might over-do this sort of project, but Jamie Jackson is aware of this possibility 
and mentioned that the timing of the project (near the end of the year) was intentional. For a 
number of reasons, including community celebration (the project included an art auction, in which 
the evaluator observed and participated). 

Student focus-eroun . The evaluator selected a random group of 9 eighth grade students 
from the membership roster with whom to hold a focus group about experiences in and 
perceptions of the Math Lab. Three students were selected from the roster for each Math Lab 
period.^' The group included 4 boys and 5 girls. Questioning for the focus group was governed 
by a protocol developed for the pmpose (see Appendix E). 

The 9 students did not, as a group, seem the least bit uncomfortable speaking with a 
stranger about the Math Lab, their school, or their community. A natural range of forthrightness 
existed, of course, with some students voluble and humorous, and others so shy that they offered 
comments only when directly prompted by the evaluator. The pmpose of a focus group is to raise 
ambiguous issues in some depth rather than to develop a representative picture of comparatively 
well-defined issues. Nine students, even when randomly selected, are far to few to provide data 
that can be generalized to the experiences or perceptions of the approximately 70 students 
involved in the Math Lab this year. 

No students in this group reported that they “hated math.” “Hating math,” of course, is a 



^'Special education students were excluded because they are not assigned to Math Lab. 



common complaint among students at this level (really it is a typical response at any level of 
schooling). One student said pointedly, and perhaps with some irony, “Math is useful— not like 
history]” 

Many of those in the group attribute their current involvement with math (i.e., their 
engagement with mathematics learning) to Jamie Jackson's instruction. Two students, however, 
noted that they were scheduled in Math Lab immediately following their “regular” math class, and 
they did not like the arrangement. When the evaluator asked if anyone's attitude toward math had 
changed, many said “yes”; these students attributed the change variously to the Math Lab, to good 
instruction, and to their perception of the usefulness of math “in the real world.” These different 
comments may just be different ways of saying the same thing. When students were asked what 
they would change about mathematics instruction at the school, the only response offered was “to 
buy more up-to-date books.” The regular eighth-grade class (also taught by Jamie) is “mostly 
book work” according to one student (with whom the others seemed to agree). 

Math Lab, some students explain: (1) “provides extra help”; (2) “gives us more 
opportunity to understand what's going on”; and (3) “helps us understand the concepts” This last 
point was made by a student who had identified math as troublesome, who volunteered very little 
during the focus group, and whose comments, when given, were not very elaborate. That this 
particular student should have made the comment most salient to conceptual understanding struck 
the evaluator as remarkable— a possible indicator of the extent of Math Lab's accomplishment 
(that is, cultivating the conceptual accomplishment of seemingly weaker math students, but whose 
improved understanding can provide the greatest boost to school and district academic 
performance). Perhaps the evaluator is making too much of this single remark, but such an 
interpretation would be consistent with other observations made at Ambler Hill. 

The students affirm that Jamie Jackson is widely recognized among students and their 



parents as a good teacher. She has shared her email address with students, whom she asks to 
contact her if they have a question (i.e., presumably a question about mathematics assignments or 
understanding). When the evaluator asks who has email at home, 5 of the 9 students answer that 
they do have it. In fact, two or three Internet Service Providers operate in the district. When the 
evaluator asks if Mrs. Jackson has any pets, the students appear to ponder this question soberly. 
However, one student says, with a charming smirk: “Well, yes, she has pet cowsl” 

Students appeared to agree that Jackson did not have pets or show favoritism, though 
several students noted that “some kids get under her skin” from time to time. In answering the 
question seriously, it seemed that the students tried to assess the extent to which the irritating 
students suffered in Jamie's classroom on account of their irritating behavior. They seemed to 
conclude that Jamie was equally concerned for all students' mathematical understanding. 

The evaluator asked one conceptual question about mathematics, just to see if any student 
would have an appropriate response. The question (see Appendix E, question 5) is “What is 
algebra?” Robert Moses has called algebra “the gatekeeper course” because students who do not 
take and succeed in algebra are unlikely to pass through the gate to advanced learning, and recent 
research by Ma and Willms (1999) affirms Moses's perception. The Math Lab, according to 
Jamie Jackson, “does algebra, but we don't call it that.” The response came quickly from one 
student, who (correctly in the evaluator's view) identified algebra as a way to find missing values 
(“finding the letters”). Other students seemed to grasp the idea of solving equations for 
unknowns, as well. 

When asked if they knew more mathematics than their parents, 7 of the 9 students replied 
emphatically that they did. One student said that “things taught now weren't taught” when their 
parents were in school,” which statement elicited agreement from others. Another student said, 
“They forget formulas,” a remark that caused the evaluator to smile inwardly. The parents of 
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these students would be younger than the evaluator, and certainly these matters were taught in the 
1960s - 1980s. Whether they were taught to so many children in Ambler Hill might well be 
dubious. 

Questions about the school and community elicited appreciation for Ambler Hill's small 
size, everybody knowing everybody else (i.e., much like Bethann Evermyer's statement about 
adults), and having all ages of students close together (i.e., on the single campus of the district). 
One student said that it was good that all K-5 students “got to play intramural sports” 
noncompetitively. Among the things students would like to see change at Ambler Hill were the 
following: (1) less picking on other kids; (2) separating kids by age-range on buses^^ (currently 
all ages ride together); and (3) “less boring stuff--we want more instruction like in Math Lab- 
doing stuff and not just worksheets.” These observations are not listed by any sense of their 
priority (not the purpose of a focus group), but the final item enjoyed wide agreement in the group 
and is salient both to the intentions of the Project and the ongoing efforts to alter the culture of 
instruction in the AHSD. 

Math Lab stained glass competition and auction . In late April, following a day of 
interviews with community members, the evaluator took up the role of participant observer at the 
Math Lab stained glass competition and auction. The stained glass works were displayed for two 
days and, it seemed, most of the classes at the K-8 building had come to view the works. The 
results really are nifty and impressive, and the evaluator is struck with their variety and polish. 

The kids have set up chairs for an outside auction, which is a bit of a problem. The day has 
threatened rain and is quite windy. But all are busy and focused, evidently excited about what is 



According to all the students, older kids pick on younger kids, but this phenomenon is almost always a 
within-family pattern, with the older kids picking on “the younger cousins.” The desire that this pattern 
be stopped, then, might reflect the uncomfortableness of middle-grade students being “picked on” by their 
high-school aged relatives, and not the concern of eighth grade students for primary-age children. 
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to come this evening. 

Jaime Jackson asks the evaluator to participate in the judging. She needs another 
judgement, she says. The evaluator does not ask why, but later learns that a three-way tie for first 
place needed to be broken. The job is simply to assess all entries according to four criteria: math 
content, creativity, neatness, overall appearance, and then to identify the best two works. The 
evaluator takes the task quite seriously and devotes perhaps 45 minutes to the judging, first 
selecting five works holistically as in the top rank, and then assessing each carefully according to 
the set criteria. This is evaluation work with no delayed gratification, and seems remarkably 
enjoyable because of the instrumental connectedness it affords the visitor. 

The evaluator chooses two works, one titled “Resurrection Cross” (nicely executed, a 
nonstandard star-like sort of cross, traditional stained glass colors, adequate math content) and the 
other called “Evening Lily,” an art-nouveau sort of design, imported from the web, with re- 
designed colors and enlargement of scale. Astoundingly, as it happens, this judgement breaks the 
tie, and these two works are awarded joint first-place prizes. 

Because of the rain, which has finally materialized, the entire event has been moved 
inside. Kids have made 10 or 20 signs directing folks through the school to the cafeteria, where 
the auction will be held. Community members begin to arrive about 5:30 p.m. 

Kids are keeping a sign-up sheet, handing out bidder numbers, and working on a laptop 
computer to keep records (quite possibly Jaime’s laptop provided by Project REAL). Jaime is 
greeting and organizing the crowd, which tours the exhibit. The inside exhibit at least does not 
have to fight the wind. Melissa Lenville sits near the evaluator tells him that the local state 
senator visited earlier in the week, saw the glass exhibit, and left a donation for a bid. Greg 
Morton, the superintendent, and many faculty are present. Morton says this is “a first” and he'd 
like to see more of this sort of thing— perhaps an academic booster club or something, just to get 
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parents more involved with the school. 

The kids have put up about 75 chairs, and add 50 more, and the room is filling up at 6:00 
p.m. as people continue to arrive. The crowd includes small children, parents, and others. The 
evaluator does not see very many “senior citizens,” however. At 6:10 there are perhaps 125-150 
people in the room, many in newly set up chairs, but many standing as well. Math Lab kids bring 
out spotlights and a tripod to show off the works going under the gavel. The Math Lab contingent 
of kids are expectant, smiling, maybe a bit tense, but perhaps feeling like they are about to be 
valued, to shine. It seems a lovely moment for them. 

At 6: 1 5, Jaime Jackson takes the podium and thanks the audience for turning out and 
explains the purpose of Math Lab, how the stained glass project was carried out (including 
research done by the kids on the World Wide Web), and what the relationship to mathematics is. 
Notably, she thanks all the parents for having such great kids. 

The bidding starts about 6:30. The auction is lively, fimny, with twists and turns, and the 
prices climb through out, with participants (including the evaluator) bidding up prices 
intentionally. The first work, one of the first-place winners, sells for $110, but the last piece to 
come to auction received the highest bid of the evening $230, from Bethann Evermyer, who could 
not be present, but for whom Melissa Lenville did the bidding. The evaluator asked Melissa what 
Bethann’ s limit had been. Melissa replied, “There was no limit!” 

In all, the sale of these 23 pieces raised $2,440. Applause at the news was loud and long. 
This event was academically relevant, in aid of the common good (75% of the proceeds were 
donated to the American Cancer Society), and an emotional celebration of the community’s 
children. The opportunity to witness such an event might be enough to help skeptics affirm the 
educational significance of a strong rural community. 

Other interactions at Ambler Hill . The evaluator conducted classroom observations in 
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mid-March and in early April he conducted interviews with most of the teachers in whose 
classrooms observations had taken place. In late April, as well, he conducted extended interviews 
with three community members. These interactions were conducted to provide organizational and 
community context to the focus of the case study, which, of course, was Math Lab. 

Observations were made in six classrooms (i.e., in addition to observations of the Math 
Lab, made on other visits) and took place several weeks after administration of the fourth and 
sixth grade proficiency tests. Each of these observations is summarized next, followed by 
summaries of the interviews. 



(1) Fred Nelson's eighth-grade science class was engaged in what seemed to be a four- 
day project of “taking notes” for chapter 1 1 in the science textbook. Fred advised the 
class they should have accumulated 30 notes by the end of the day. The goal is to 
assemble 100 such notes by the end of the week. Indeed, it is not at all clear to an 
observer exactly what constitutes a “note” to be accumulated. No standards or 
rubrics are on display for what seems to be a common activity. Fred circulates 
during the entire class and interacts easily and respectfully with students, who behave 
well throughout the period. There is some evidence of other activities: an incubator 
project is ongoing and Fred confers with one student about the identification of a 
bird, using a field manual. Posters on the wall concern science content. 

(2) Susan Ferris's sixth-grade math class is apparently a self-contained gifted class. A 
visitor gets the impression that these children are of various ages, but the lesson is 
the same for all students: equivalent fractions. Susan explains how to perform 
cross-multiplication to obtain missing numerators and denominators. The lesson 
does not include explanation or discussion of the logic of equivalency, or in just 
what sense fractions might be considered as “equivalent.” Students work at the 
board (displaying previous calculations) and Susan helps the group examine the 
work, mostly in monologue. Following this event, Susan assigns textbook work and 
says that students may work with a partner. Susan tells the evaluator that this is her 
first year teaching in a regular position, that she holds a masters degree in gifted 
education, and previously worked as a substitute teacher. 

(3) Denzil Kline's seventh-grade science class is seemingly begirming a project on 
weather. The discussion feels to an observer, however, as if previous instruction 
about weather has taken place, and today's lesson is an introduction to the 
instruments students will be using and to the measurements they will be taking with 
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the instruments. The class is lively, with the Denzil asking rapid-fire questions of 
students that relate science measurement to mathematical understanding, that call for 
logic and, to some extent, the giving of reasons for answers offered^^, and also 
includes an impromptu review of procedures and terminology for this project. 
Students respond freely, and Denzil seems to distribute his questions across a range 
of students. 

(4) Melissa Lenville's sixth-grade class^'* is having a lesson on probability, and the 
students are doing “tree diagrams” based on two- and three-step draws comb inin g 
elements to generate permutations. Melissa circulates in the classroom as the 
students work on assigned problems, and as several students finish their work, she 
dispatches them to help students whose hands are raised. This lesson exhibits 
comparatively little discourse about the concepts or applications of probability. 
Students seemed to grasp the operations involved comparatively well, however. 
Above the black board at the front of the room is a number line, sporting both 
positive and negative integers, and a followup question to Melissa elicits the 
information that she does teach negative numbers, and the number line is an aid in 
that effort. 

(5) Hanna Dodd's fourth grade class is joined next by the evaluator, who enters during a 
story segment, with Hanna reading from the C. S. Lewis “Narnia” stories. The 
classroom walls are packed with language arts content-displays, student work, 
posters, and so forth. When the reading ends, Hanna talks to the students about the 
story and then segues into a science lesson (environmental education). She uses a 
CD-ROM visual and conducts a discussion about the concept. The mantra of the 
lesson (repeated several times-though the evaluator is uncertain he has actually 
gotten it right in his field notes!) is “revise, re-use, recycle.” Students are quite 
engaged in the lesson and offer a variety of observations and comments. 

Interestingly, the video clip, which is the center of visual attention, is (quite 
appropriately in the evaluator's estimation) more of a prop and a prompt than a 
source of information, and the actual footage viewed lasts less than a minute in a 20- 



The consideration of humidity is particularly interesting to an observer. The students have difficulty 
with the concept and Denzil tries to explain the idea of relative humidity. The salient question, of course, 
is “humidity relative to whatT’ Denzil explains that this means “relative to how much water the air can 
hold,” but an observer could not be certain that the students grasped the idea that warmer air has the 
capacity to hold more water than cooler air. This idea might be worth a full lesson in itself. The frank 
encounter with this complexity, however, even if not perfect, was at the very least a good start for 
students. The application of ratios is critically important for students of this age, and they commonly 
receive few opportunities for such application. 



In the era of new math the evaluator personally did not encounter concepts of probability until the 
eleventh grade. 
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minute class segment. 

(6) Teresa Cotton's fourth-grade science lesson is the last class observed.^^ Teresa is 
teaching what appears to her visitor to be an unusually imaginative, participatory 
lesson, with episodes of both quiet reflection and active production and interaction. 
Teresa assembles the students loosely in a circle, directs them to imagine they are an 
animal or human, wild or domestic, and then proceeds to establish a mood. She 
begins with a teacher's manual in hand, but soon abandons it. The mood is 
established with several props besides her instructions: music, pie tins for thunder, 
and a spray bottle for rain (the students receive a light spray from this prop, just as 
the story narrates rain). The ideas here are habitat, shelter, and also erosion. Teresa 
seems to take a detour to the last idea as she asks students if any of them were a fish. 
How would rain affect a fish? How would run-off affect a fish? Could it be life 
threatening? Teresa allows the students to struggle a bit with this situation. In any 
case, practically all the students offer a story about the creature they imagined 
themselves being, and the consideration of “the fish problem” is rather lively. 
Another active part of the lesson entails students creating a six-panel illustrated 
narration of their imagined adventure to include a comparison of the real student's 
need for shelter and the need of the imagined creature. This part of the lesson cannot 
be concluded today, and Teresa has an orderly procedure for storing work to be held 
for completion in the future. 



The classroom observations at Ambler Hill elicited a range of sample practice related to a 
more active sort of instruction in most classrooms; one might even say, “virtually all classrooms.” 
Even in the least imaginative lessons or classes observed, glimmers of hopeful inroads against 
utter de-contextualization and student disengagement seemed evident to the evaluator. One of the 
classes seemed intellectually dreary, certainly, but the easy quality of classroom relations did not 
suggest to the evaluator that academic content was being misused as a route to achieving an 
orderly classroom— there was no such misbehavior in need of control. The class was just simply 



“ There is some misbehavior during the lesson, and Teresa stems it easily with a kind of low-key time-out 
ritual. In interviews conducted subsequently, Teresa reports this misbehavior was highly unusual and she 
cannot really account for it. Nonetheless, in conversation she discounted the influence of the evaluator's 
presence and speculated that the misbehavior might have been the result of one child's failure to take 
prescribed medication. Bethann Evermyer had praised Teresa's commitment to effective classroom 
management, prior to the observation, and it was useful for the evaluator to see Teresa's methods in 
action. Her voice exhibited firmness and not anger or frustration, and the “response cost” administered 
was not more than was necessary to control the misbehavior. 
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not designed to engage students authentically in any way. 

In the evaluator's estimation two of the lessons observed were very much in the “hands- 
on” mode, and two others exhibited strong symptoms of student engagement and, possibly, self- 
conscious teacher growth. In the other class, a new teacher seemed to be struggling toward 
improvement and making headway. The conclusions from these observations may be all the more 
remarkable because these were not announced visits. Conclusions are necessarily tentative, but 
are intended more as a portrait of the instruction at this school on this day than of the instructional 
practice of any particular teacher. 

The evaluator returned in early April to interview some of the teachers observed. The visit 
opened with a long implanned interview with Bethann Evermyer, who told the evaluator of the 
new arrangement for planning time just implemented with the approval of the faculty. Bethann 
had imagined this arrangement over the weekend, sought coimsel with Greg Morton, the 
superintendent, and with Jamie Jackson, and then asked the teachers if they were willing to try it. 
They were. 

The new arrangement was to provide four 135-minute planning periods every three weeks. 
It also was so arranged that teachers would have common planning time. The previous schedule 
provided each K-8 teacher with three 30-minute planning periods per week. The new 
arrangement increased the total time available by 50%, but consolidated it into a block sufficient 
to permit both collaboration and extended effort. The field notes assert that “Bethann is nervous 
and hopeful about whether the change will be accepted, and the teachers themselves are skeptical 
but willing to try.” 

We also talk about departmentalization for the lower grades. Bethann states that she has 
personally always been opposed to it for the elementary grades. “Golly,” she says, “You don't 
need that much ‘specialized knowledge’ to teach even seventh and eighth grades.” This is an 
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interesting opener to what follows, because it turns out that Bethann is rather supportive of Teresa 
Cotton's willingness to become the fourth-grade science specialist. This change is possible in the 
coming year. The outlook for math specialization is not so rosy, but Betharm indicates her 
willingness to try that, too, should the opportunity arise. Betharm says that she appreciates the 
idea that what works is what teachers want to do and are good at doing, and Teresa she thinks is 
(according to the field notes) “distinctly fired-up about hands-on science.” The evaluator's 
classroom observation notes would seem consistent with that judgement. 

The teachers at Ambler Hill offered the following remarks in response to the evaluator's 
query about the best they could say about the Project: 



Team member : I've actually learned a lot of hands-on methods. Even [the most 
“traditional” teachers] are doing some of this stuff. 

Team member : Project REAL is the most thrilling, engaging, and exciting, discovery- and 
project-based program I've experienced. What I appreciate most is that it treats the how 
and why of learning. 

Team member : I'm more relaxed about teaching and learning, and more willing now to 
leave the [science] textbook alone. And the same goes for math. REAL has really 
changed [my colleague's] thinking and has carried over to [another colleague, not a team 
member] as well. I'm 99% positive on the Project. 

Team member : Project REAL is the most thrilling, engaging, and exciting, discovery- and 
project-based program I've experienced. What I appreciate most is that it treats the how 
and why of learning. 

Teacher who is not a team member : Hopefully we will get ideas to teach different 
strategies and areas. Teachers [at this school] are more than willing to do what it takes to 
see students succeed. 

Teacher who is not a team member : The Project is encouraging more hands-on lessons, 
more student involvement, less book- and seat-work, more relevance to the real world, and 
improved Proficiency Test scores. 

In response to the evaluator's query about the worst they could say, the teachers (same 
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order as above) replied: 



Team member : There was a lot crammed into the week of REAL training last summer; 

there was an awful lot of lecture being told to do hands-on. 

Team member : We don't have the materials we need for problem-solving and real-world 

activities. 

Team member : The time it takes, because it all means being away from the classroom. 

Non-member : I have heard nothing bad about the Project. 

Non-member : One of the obstacles to the Project is a lack of teamwork. 

Denzil Kline and Teresa Cotton, both members of the local Project team, gave perhaps the 
most “hands-on” of the lessons observed at Ambler Hill. In addition to Jamie Jackson and Beth 
Evermyer, these two teachers appear to the evaluator as likely instructional leaders. Interviews 
with them are summarized more fully, below in order to profile their contributions to the AHSD 
organizational culture. This report of information about the ongoing implementation of the 
Project begins with them. Data from the formal interviews with Denzil are augmented by data 
from an informal conversations conducted following the classroom observation reported above. 
Incidentally, both Denzil and Teresa grew up locally and graduated from the AHSD high school. 

Denzil has his lunch period after this class, and he and the evaluator talk about the class 
and teaching in general. Denzil says flatly that his involvement with Project REAL has motivated 
him to change the way he teaches. (In another conversation, Betharm Evermyer confides that she 
believes Denzil to be the “most changed” teacher this year at Ambler Hill.) Denzil holds an MA 
in educational technology, previously taught physical education for six years, and has taught 
science for the past two years. “I'm learning with the kids,” he says. He says that his “big 
problem” in teaching this way-which he thinks the kids like better and which benefits them more 
than his former instructional style-is “creativity.” He doubts he can come up with lots of 



activities on his own, and feels the need for a guide. 

Denzil, however, simultaneously talks about constantly changing his lessons, tweaking 
them for improvements, experimenting with new ideas— in a word, “developing” instruction and 
curriculum to suit his purposes and capacities. He appears to be doing this under his own 
authority— acting autonomously in a professional manner— but with the strong encouragement of 
Project REAL (by his own report). Denzil lives in the district and has a child who will begin 
school next year in the district. 

During this conversation, Denzil confides to the evaluator that being observed made him 
nervous. The comment is remarkable for two reasons: (1) this was an excellent lesson in the 
evaluator's judgement, with Denzil very clearly holding the attention and interest of the entire 
class (in the period just before lunch, at that) and (2) making this comment to a stranger indicated 
a high level of candor and security. Denzil, in fact, would probably not have appeared 
uncomfortable to anyone observing his teaching of that lesson that day. 

In the interview, Denzil, addressing issues of change and resistance, reports that “the two 
older teachers are getting more involved— doing more the past couple of years.” Denzil mentions 
the incubator project, for instance, in Fred Nelson's room. The fact that Denzil mentions this 
activity unprompted by the evaluator perhaps lends validity to the claim that “hands-on” methods 
are widely evident at Ambler Hill. Fred, as he himself probably knows, serves as the icon of 
“traditional teacher” in Ambler Hill. Change of any sort in Fred's teaching is regarded by Denzil, 
and perhaps others, as notable. Fred’s teaching practices are perhaps quite resilient. 

Denzil reports that there is “spill-over” into the lower grades. He reports that more 
teachers are coming to him for ideas, asking such questions as “You got this? What can I do?” 
Denzil reports, for instance, that he interacts with Fred on such a basis. 

Denzil has high praise for the Project, and when asked what is the worst he might say 



about the project, he observed that the summer 1999 training was very full and not very hands-on 

in its approach.^® In any case, Denzil reports that the best thing about the Project is that he 

actually “has learned a lot of hands-on methods.” The evaluator's field notes contains this 

observation about “the best” (learned a lot) and “the worst” (lots to learn and not very hands-on) 

as interpreted by Denzil in a formal interview: 

This response is a one-two blow, almost making the same point, or at least 
demonstrating how the bad is integral to the good, and how people are necessarily 
making this stuff up as they go along, classroom teachers as well as trainers and 
those theorizing about constructivism; which is not surprising given the 
commitments of dyed-in-the-wool constructivists: improvisation is the name of 
the game, after all. 

Teresa Cotton also harbors a very good opinion of Project REAL. She states: “Project 
REAL is a real motivator in my case for hands-on and project-based learning.” Commenting on 
the prospect of becoming the fourth-grade science specialist, Teresa says she like it. She enjoys 
teaching science, she says, “because I can pull everything together with science— math, writing, 
reading, problem-solving, social studies, art.” This view is certainly borne out in her teaching 
behavior during the lesson observed, in which divergent thinking, oral communication, writing, 
social studies, as well as art and music were all in evidence. 

This reason for specializing strikes the evaluator as surprising. The reason given exhibits 
an intellectual freight, and does not concern either teacher convenience or the ease of planning for 
one subject— the usual reasons mentioned by teachers.^’ Most surprisingly, however, this 
reasoning uniquely (and, by inference, complexly), gives wholism as a reason for specialization. 
This interchange alone suggests to the evaluator that Teresa engages teaching reflectively and 

^®The evaluator wonders if the challenge to create more engaging, active classrooms may be greater for 
higher education than for lower education. 

^Another teacher interviewed at Ambler Hill, in fact, did offer these more usual reasons as possible 
arguments for specialization. 



with considerable intellectual depth. Teresa's vantage on the Project is very positive (her remarks 
were quoted above). 

Responding to the query about the worst that can be said of the project, Teresa had 
referred to a shortage of relevant materials. This remark more likely constitutes a criticism of 
local Project decision-making than of overall Project operation, since decisions about how to 
expend Project funds are purposefully reserved to demonstration sites. The evaluator asked, 
probing, if Teresa had Internet connectivity at home (it seems that Teresa would find materials 
and activities she could readily use in her classroom on the web); she responded that she did not, 
and that she has set two goals for herself for the coming year: (1) to learn more technology and 
(2) to get better organized. When the evaluator asked if she were a goal-directed person, she 
grinned and said, “That's just the way I am.” This is evidently a strong personality trait. 

The evaluator is interested in the teacher-leader role that Teresa must play, and so he also 
asks about her future career plans. She is planning, and her plans center either on school 
psychology (which she doesn't favor all that well because it would remove her from the 
classroom) or “something to do with helping teachers.” The word “supervision” does not appear 
in Teresa's description of this work, and even when it is mentioned, she is unsure what it means 
(in the context of schooling). So the evaluator explains that “supervision” is what “helping 
teachers” is most often called in school jargon. 

“Kind of,” he says, “like what Bethann does.” 

“Oh yes,” says Teresa, “Bethann is on our side\ she's doing that for us [emphasis added].” 

The phrase “on our side” seems telling in this district. Teacher leaders seem to feel their 
efforts are supported and their perspectives appreciated. They see that support as withing 
Bethann’ s role more than the superintendent’s, whose efforts are directed elsewhere, perhaps 
toward the high school and interactions with the school board. 



The evaluator senses a high level of appreciation for the Project from a diverse group of 
faculty at Ambler Hill. The misgivings expressed seem to concern the problematic nature of any 
sort of change, rather than the design or operation of the Project. Faculty articulate to themselves 
and to each other their aspirations for better teaching and learning. Team membership has been 
stable since the Project application was made. 

Teachers shared their view of barriers to and facilitators of organizational change. 
Evidence exists that the school is addressing at least one of the perceived barriers (perceived lack 
of collaboration). According to teacher interviewees, however, the barriers include the following: 

• lack of collaboration, 

• lack of materials, 

• personal shortcomings (many associated with individual plans for improvement), 

• misalignment of curriculum and instruction, and 

• negative feedback from administration (meetings, policy, union negotiations). 
Facilitators were, however, comparatively more numerous than in the other case-study 

district: 

• Project REAL (mentioned positively by all interviewees), 

• pre-service and inservice training, 

• supportive supervision (Betharm's leadership), 

• good students (hard working, etc.), and 

• good colleagues (hard working, caring, love teaching, etc.), and 

• status of school as community center. 

The evaluator asked Melissa Lenville, who served as the school secretary prior to 
begirming her career as a teacher, what the center of the community was. She responded that it 
was the sports fields— football and basketball. When the evaluator replied that a lot of 



commentators on rural schooling bemoan the role of sports, she observed that “those people don't 
seem to realize that a lot of time, it's involvement with sports that keeps kids in school. But,” she 
adds, “we need to have teachers who are coaches and not coaches who are teachers.” She appears 
to mean that academics has to be a clear priority, and that good teachers extend their classroom 
influence as advocates of learning when they also coach. 

Selecting and reporting community members’ perspectives about schooling . The evaluator 
asked four or five Amber Hill educators to nominate community members who could provide 
insight into the nature of the community that Amber Hill serves. Bethann Evermyer supplied 
contact information for these people, whom the evaluator contacted personally to arrange 
interviews. Guiding questions appear in Appendix F. 

Numerous calls subsequently yielded scheduled interviews with three people: Abe 
Franklin (a local bank official in the declining industrial town within a half-hour’s commute), 
Brenda Martin (a fulltime mom, who, with her spouse operates a car dealership in the same town), 
and Becky Frye (who is not working and whose spouse has just been laid off due to a plant 
closing— from the same town in which the family of the other two interviewees draw their 
incomes). The evaluator asked for people willing and able to talk at length, but not just 
spokespersons for the middle-class, often nominated in such instances. 

It seems he was partially successful, but, it is still important to note that none of these 
people is disaffected from the school or the community. Two— Becky and Brenda— are natives of 
the area within commuting distance of the school. Abe was raised in a rural community in 
another part of the state and has been relocated several times on account of his career. Brenda, as 
it turns out, was raised and, until recently lived, in the town where her family’s business is 
located; her spouse’s family, however, farmed within the district and they recently moved to the 
district, partly so their children could attend this school. 
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These three community members represented their worlds uniquely, and with gentle 
pathos and passion. Conversation with these people was easy and took the full time allotted in 
each case. They offered frank opinions about the school, about educational purposes and 
methods, and about their educationally relevant commitments and worldviews. They were 
articulate and less guarded than some of the educators interviewed (community members arguably 
have less at stake in these interviews than educators). 

The report that follows attempts to preserve three separate stories rather than to present a 
synthesis that might falsely imply consensus or general applicability. Uniqueness is more 
important than commonality here, and the evaluator’s object was to elicit some of the rural 
meanings surrounding the life of the school and, more particularly, school reform in a rural locale. 

Community members’ stories . On the day scheduled for interviews vsdth community 
members, the evaluator arrives a bit early and watches the hall traffic outside the office. Teachers 
are placed at every junction to observe kids; gently they remind kids to walk, not run, et cetera. 
Adults smile and chat with kids; some kids do look troubled or vexed as they hurry about, but the 
level of tension feels low in comparison to some schools the evaluator has visited. 

One kid is standing near a teacher, apparently under some sanction with her, but he seems 
to take it in stride. Conversation turns to the upcoming vacation (this is the last day before spring 
break). “Thank God,” says the kid, “I get to visit my dad.” Another teacher (or aid), familiar 
from previous visits, approaches the kid at this juncture and says, “Yes, you'll get to see your 
colt.” And they chat about this. The visitor asks about the colt— bom in Febmary, says the kid. 
Then the evaluator remembers that he has a meetings and discovers that Abe Franklin, the first of 
three interviewees today, is already sitting in the conference room where most of the interviews 
with teachers have been conducted. 



First Community Interview 

Abe Franklin grew up in a poor rural county in Appalachian Ohio. Now a widower, he 
has lived here for eight years, and his son is concluding his senior year at the high school. 
The bad thing about working for the bank, recalls Abe, is that “we never really had a 
home.” The bank transferred him to a new location four or five times diuing his 35-year 
career. Though an increasingly common experience for most Americans, Abe feels it 
denied him a home. 

When he retires (seemingly in a year or two), Abe imagines he will return to where he was 
bom and raised, which is also where he and his spouse began their family. “That,” he 
says, “is where my roots are.” He tells the visitor that he’s been a widower for several 
years. 

Abe says that although not from around here, he imderstands this rural commimity’s hold 
on his neighbors. “They really want a way of life that's not possible any more; they want 
to keep what they have; and they are maintaining their values” [emphasis added]. His 
remarks are more in the way of observation than criticism. In fact, given his retirement 
plans, his own aspirations are quite consonant with those of his neighbors. These two 
thoughts held simultaneously suggest that Abe is a man who appreciates complexity and 
contradiction. This impression grows during the course of the interview. 

Though his grandfather reportedly “taught school,” Abe was the first in his family to 
complete college. Abe confided that he “hadn't really planned to attend college,” but a 
close friend said that since he was going, Abe should come too, and they could room 
together. This is a pleasantly minimalist, laconic, rationale for attending college. The 
contradiction is intensified as he reports that his friend, ironically, left college early to go 
to work in a family business, but Abe persisted and graduated. 

After graduation, he says, “I took the least well-paid of four job offers.”Abe muses that, in 
fact, he’d made the best career choice among the options presented at that time. He’s 
enjoyed his career, he said. He seemed secure and thoughtful to the evaluator. 

He says that he has insisted that his son (the last of three children at home) earn 
“reasonable” grades — A's and B's expected, with an occasional C tolerated. His son, he 
says, is not a good scholar and doesn't want to go to college, and will probably work 
construction for the father of one of his (the son’s) friends at the high school. “Hell make 
pretty good money,” predicts Abe. “Maybe he'll surprise me though,” he adds, “and 
decide that he wants to return to school.” Abe says he is perfectly comfortable with his 
son’s decision not to attend college. Later on in the interview, Abe confided that his son 
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had “passed” all Ohio Proficiency Tests at both the 9th and 12th grade administrations. 

On the basis of these remarks, one might logically infer that criterion performance on the 
accountability tests is not, in Abe’s view, sufficient to establish “good scholarship.” This 
is probably not a widely shared view, since such a performance has become a prime object 
of Ohio schools. 

Abe says the quality of instruction here is not good; and that expectations are low (a 
paraphrase the evaluator again confirms with the interviewee). The evaluator asks if these 
remarks apply mostly to the high school, and he says, “Yes.” 

“I think the teachers all care; want students to do well; and they cultivate discipline. They 
want to see students become good citizens, I guess.” Abe reflects on the matter and says 
that he thinks many of what he sees as shortcomings have to do with the nature of local 
circumstances, which seems to refer to dimensions of local culture. “People around here,” 
he says, “don't think about leaving, don't want to leave. Out of the kids in my daughter's 
class, only two went to college someplace besides local colleges. (His daughter went to 
Cincinnati, and so was one of the two.) This afterthought, again, seems less a 
condemnation of this sensibility, and more like an observation of difference between it and 
his own view. He seems to understand and even appreciate a view that is not his own. 

One might obviously conclude that it is not the view in which he’d have preferred his own 
children to deal with in their schooling. 

Abe reports that he has never discussed math or science with an Ambler Hill teacher or 
administrator. (Given the responses to this probe from the other interviewees, the 
evaluator believes that Abe may have discussed his son’s math or science performance 
with his son’s teachers; such hypothetical discussions were, in any case, if they occurred 
not sufficiently memorable to prompt an affirmative response.) 

Abe nonetheless reports that he has “great respect” for Greg Morton. According to Abe 
he's honest; he addresses issues (rather than ignoring them); and he's friendly. Abe 
expands: “Some people say he's not [friendly], but I think he's maybe just. ..aloof That's 
his personality. You need to make a bit of an effort.... Some people resent his salary, but if 
the district is not paying $65-70K, they won't get the quality they need in a superintendent. 
I think he does a fine job.” Such a salary, one must observe, is low for a successful 
administrator in today’s labor market, and the resentment is a gauge of the economic 
straits of the area; it probably is the highest paid position in the territory served by the 
district. 

When asked what makes a good school, Abe insists that “I'm from the old school; I think 
three things are important.” These are: (1) parents who support students and attend 



school events (“my parents were like that”); (2) rewards that are commensurate with 
achievement (a paraphrase whose aptness the evaluator confirmed with the interviewee); 
and (3) teachers who set an example, who are addressed as “Mr.” and “Mrs.,” and who 
dress as professionals. In later conversation, Abe adds a fourth stipulation: (4) give the 
basics priority. He also flatly endorses the concept of social promotion: “Why hold them 
back if they can't do really well?” 

Responding to the question about what makes a good teacher Abe says “Well, that's 
related to [what the good] school [is].” (1) teachers should hold the respect of students; 

(2) they should have good subject knowledge; and (3) they should exercise strong 
classroom control. Abe approves of judiciously given corporal punishment— but for boys 
only. Because his youngest son is just completing high school, his remarks almost 
certainly pertain to the high school. Later, he confirms this is the case. 

Abe reports he has not heard about Project REAL before today. He does say that Jamie 
Jackson is an excellent teacher. His son, he says, learned more from her than from any 
teacher ever. “But,” says Abe, “something seems to crumble in high school in this district. 
May this is a common thing in other districts.” (This observation, in fact, does accord 
with the evaluator’s experience elsewhere, see for example, Howley & Harmon, 2000). 

Abe is not sanguine about the form of instruction sponsored by Project REAL. “I think 
we've tried hands-on learning, and it looks like it's not as effective as was expected,” he 
says. “Students laugh at [hands-on activities]; they think it's a waste of time,” he adds. 
The evaluator, again careful to determine context, asks if Abe is referring to high school 
students, and he again says, “Yes.” 

At this juncture, the evaluator improvises a challenging question for the interviewee. On 
one hand, Abe had said (at one point in the interview) that “you'd think schools would be 
more advanced than what they were” when he and the evaluator were kids, 30 or 40 years 
ago, but, Abe had indicated that he thought that schools were “pretty much the same now 
as then.” And on the other hand, he not only favors giving “the basics” priority but he 
does not endorse “hands-on learning.” 

The evaluator asks “Do you see the tension in those views? Or can you explain how you 
view that tension?” This is tricky stuff, in the evaluator’s view, and he imagines that Abe 
Franklin can sense this is not a hostile question. 

He says, “Mm. Well, I think the schools should be doing more with speech— with 
communication— and computer technology. I'm not the world's best communicator, we 
didn't address speech or communication when I was in school. It's very important when 
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you're working.” Had we the time to discuss the matter further, Abe might have had a lot 
more to say on this point. Nonetheless, he easily tolerated a level of ambiguity and 
contradiction that makes many people uncomfortable. 

The chain of remarks in this interview struck the evaluator as describing an approach to 
life with a certain amount of Zen (not precisely Appalachian fatalism, but something akin 
to it)— ultimately finding one’s place within decisions, rather than forcing or hurrying 
them. Such a disposition in a banker surprised the evaluator, whose experience of bankers 
is that are decidedly literal-minded. Abe is very different: decided in his views, but 
recognizing not only the give-and-take of living, but the give-and-take of thinking. 

Second Community Interview. 

Brenda Martin is the current president of the Ambler Valley Parent Teacher Organization 
(PTO). The Martins, as noted above, are originally from the nearby declining industrial 
town. Brenda went to school there, but spent her summers and holidays on her 
grandparents’ farm in nearby West Virginia. The Martins have two children and a third 
obviously on the way, due in July. Their son is now in the sixth grade, but has attended 
the Ambler Hill school for three years, under the terms of open enrollment. He is the elder 
of the two Martin children, and the evaluator infers that Brenda’s remarks, unlike Abe 
Franklin’s, refer to the K-8 building, especially the middle grades (the focus of this 
evaluation). 

The Martins recently moved to the farm that they bought seven years ago, located in the 
Ambler Hill attendance area. Brenda did not like the town schools and thought they were 
badly run, so when their son began middle school, they transferred him here, and drove 
back and forth twice a day for two years. The change has worked well, she says, and they 
“love the school.” Later in the interview she observes, “There's no prejudice out here,” 
and few cliques in the school. The remark appears to assert the absence of race 
(“prejudice”) and class bigotry (“cliques”). 

The evaluator, whose research specialization concerns school size, questions Brenda about 
the issue. Large size was one thing, she says, she did not like about the town school. “I 
feel smaller is always better.” The evaluator makes the neutral observation that this 
particular rural school in Ambler Hill enrolls about 700 kids in grades kindergarten 
through eighth grade, and he claims, “that’s not small.” Brenda looks startled, or perhaps 
embarrassed— it’s difficult to judge. 

She stops talking momentarily and considers the problem. She says, still puzzled, “Yes, I 
always think of a country school as small.” But in fact, the town schools were smaller 
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(though each was segmented in narrower grade span configurations). The town high 
school, in fact, is not much larger than the Ambler Hill high school. Apparently, to some 
of those involved, the Ambler Hill school feels small. 

What makes it this way? Many of the teachers say that they think of the K-8 building as 
three separate schools-primary, elementary, middle. So the evaluator asks Brenda if she 
thinks of this as one school or three. She says “one.” The evaluator, also outside the 
professional community of the school, also thinks of it as a single schools because it has a 
single principal and exists imder a single roof.^* 

Brenda knows the teachers here, partly through her involvement as a kid in coimty 4H. 

She was, she says, always involved in farm life. Her grandparents, as noted, had a farm, 
where, she reports, “we spent weekends and all our summers.” She felt her son was 
“vulnerable” in the town school and would be easily persuaded to do things he should not. 
He adjusts well, she says. Brenda seemed to appreciate the idea that adjusting well was 
not necessarily a good trait. At Amber Hill her son participates in baseball and football 
(the evaluator infers these activities are organized by the school). 

The kids our here she says are all mostly “good farm kids— poor, on welfare, but very 
respectable.''’ I ask what she means. “Well,” she says, “they have maimers, they're 
friendly, they're respectful.” 

The coimtry, she continues, is magical. This is the language the evaluator’s daughter uses 
to describe her rural upbringing, and he shares the coincidence with Brenda. “The simple 
life is best,” she says, and she compares it to a life centered on acquisition and expressed 
in material terms. This sensibility is very consistent with what many commentators have 
written about as the value of a “properly rural” education (for instance. Berry, 1990; Haas 
&Nachtigal, 1997; Theobald, 1997). 

For this reason, the remark strikes the evaluator forcefully, and he asks, “How did you get 
this way?” 

Brenda blinks, surprised to be asked such a question. The evaluator senses that a 



An acquaintance of the evaluator’s, who knows the AHSD, holds the view that the school is large and 
poorly designed. It is large by contemporaiy standards, especially since it serves a veiy impoverished 
community. Yet, the evaluator believes the school is beating the diminished odds that larger size imposes 
on it. Test scores (e.g., in comparison with the much more affluent Ambler Valley district) would seem to 
give strong support to this hypothesis. Ambler Hill is for some reason an exception to the rule. Perhaps 
the teachers are right. Under Bethann’s leadership there may be three schools in one. The arrangement 
might be less successful with different leadership; Bethann reportedly helped design the building. 
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meaningful educational experience may inform Brenda’s remark, which seems quite 
confessional. 

“How did I get this way?” She pauses briefly, knowing exactly what is meant and 
seemingly wondering where to start. 

“I changed after I had kids. When I had my first child, Seth, I wanted him to have the 
'best' of everything. I put us in debt! My mom preached at me, and my connection with 
church taught me something... I guess. I've matured and grown up.” A bit later in the 
interview she observes that her mom is “this way too.” Her mother works at a sheltered 
workshop for $16K a year, but has a masters degree and could make a lot more as a 
teacher. But, she says, her mom loves the work and feels she's doing a lot of good. 

The evaluator thinks: “This is a thoughtful and humble person who speaks of her own 
growth with insight, and, moreover, connects it to a principled critique of the world. 

Could we wish for better outcomes of an education? Wouldn’t the world be a better place 
with more people educated in this way?” In any case, the transformative educational 
experience for Brenda did not happen in school.^’ 

When the conversation turns to changes in the district, with which she was familiar, 
though not a resident, Brenda remarks that the district has changed through consolidation, 
which engendered “hard feelings” initially. 

Brenda describes a good school and teachers as (1) establishing a caring atmosphere and 
(2) exhibiting “the will not just to teach, but to reach, kids: to make school feel like a 
second home, not a 'have-to' situation; that about sums it up.” Her use of the word “will” 
is striking because it so seldom appears in the discourse of educators. Indeed, it exhibits a 
distinctly “theoretical” flair. 

Greg Morton is a former teacher of Brenda’s. “We call him strict, and a very good 
Christian.” Greg, she says, or perhaps implies, knows local circumstances well enough to 
make good decisions. I ask, following Abe Franklin’s observation, if she regards him as 
“aloof” “Aloof? No,” she says. “I think,” she adds, “he has the guidance of the Lord.” 
She contrasts Greg’s leadership with the leadership in the town district. She is particularly 
displeased, but not sxuprised, to learn that the town district is re-hiring retired teachers 
only to coach; that is, without requiring any teaching assignments. In Ambler Hill, she 



Growth and insight of this sort would be regarded widely as admirable across sharp differences in 
worldview. The AHSD would seem to be quite fortunate to have patrons willing to bring such wisdom to 
bear on the challenge of running a school system. 
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says, there is no coaching without a teaching assignment. The observation echoes Melissa 
Lenville’s observation that the district needs “teachers who are coaches” and not the other 
way roimd.^° 

Brenda has not previously heard about Project REAL, nor she reports, has she ever 
encoimtered to opportunity to discuss mathematics or science with local educators. 

One of the evaluators questions concerns interviewees’ preference for textbook-based or 
“hands-on” learning. Brenda chooses the latter without hesitation. She illustrates her 
preference by referring to her son’s recent experience. The Martin’s son is in Susan 
Ferris’s self-contained gifted class, which, Brenda reports, was conducted quite 
traditionally for the first six weeks of school. After that, Susan’s teaching, according to 
Brenda became less traditional. Her son is now “doing better” and Brenda attributes the 
change to the change in Susan’s approach to instruction. Older folks in the district, 
according to Brenda, prefer textbook-based instruction on the basis of their own passage 
through school.^' 

Third Community Interview. 

Unlike the previous two interviewees, Becky Frye is a native of the Ambler Hill locale. 

She says, “We're almost 10 yrs behind everyone else-but that's a good thing; we're a sort 
of slow-to-change place.” Becky has worked as n x-ray technician for 17 yrs, but is not 
working now. She attended a local imiversity for one year and then took technical training 



Some districts, apparently, believe the functions should be separated. Why might this be the case? 
Melissa and Brenda appear to believe that such a practice harms the principal educational mission, which 
concerns academic, emotional, and spiritual growth (the latter perhaps especially significant in 
communities like those in which the Project works). More formally, and hypothetically, the practice of 
separating the two functions of coaching and teaching first of all privileges employees who prefer 
coaching to teaching. More abstractly, this action tends to privilege coaching over teaching. Moreover, 
the traditional connection between coaching and school administration— common in many areas, not just 
in rural areas— combined with such a practice could deform organizational culture to the point that 
academic performance suffers. 

The evaluator hears this assertion often, but does not understand the logic of the claim. Does it indicate 
a nostalgic view of the past merely, or does it relate the apparently distressed condition of the 
contemporary world to an imagined, more perfect past (that is, to a mistakenly imagined past)? Worse, to 
what extent are we all susceptible to this reconstruction, as part of the human condition, seeing our own 
bodies weaken, and naturally enough associating old routines with our old corporal glow? In fact, of 
course, “hands-on” instruction has been in the professional educational air since before 1 900, and was, 
albeit it in a substantially altered form, received pedagogical wisdom from the through the 1960s. 
Progressive education even of that sort, however, was probably not common in the schools of 
Appalachian Ohio. 
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for a year. Her spouse was very recently laid off from a factory in the nearby town, as a 
result of its shut-down. “We're not leaving,” she says, “we'll make our lives work here 
somehow. Im' not sure if I'll go back to work, but he won't leave, so even if I wanted to 
leave,” she says humorfully, “I'd have to do it alone. We like it here. This is our home. 
We'll be OK.” 

Since she stopped working, says Becky, she has been very involved in the school. Her 
eldest child is in the eighth grade (and so would be a student of Jamie Jackson this year). 

“I have 2 children, plus a nephew, and I drive them to school because I've spoiled ‘em 
rotten.” Initially Becky was concerned about “dual routing,” the practice of busing high 
school and elementary kids together on same route (a cost-saving measure initiated by 
Greg Morton, as Greg told me in our initial meeting); but that really hasn't turned out to be 
a problem. Bus drivers, she says, actually separate older and younger kids in separate 
parts of bus. 

Speaking of her own experience at Ambler Hill, Becky says she was a good student and 
that she did fine (all A’s and B’s) in her postsecondary work. She has one “straight-A” 
kid, she says, but “the other struggles just to get B's and C's.” As a result, she seems to the 
evaluator to harbor a degree of empathy for kids who struggle academically. “When I was 
in school you got it or didn’t, but today they are trying to teach all kids.” She 
spontaneously adds, “Jaime Jackson is a wonderful teacher— has got others into this.” 
Becky seems to mean that Jamie is helping other teachers teach more like she does. 

Like Abe and Brenda, Becky has not previously heard of Project REAL. A good school 
according to Becky is one that (1) has lots of discipline; “This one,” she says, “does”; (2) 
has 64 Jamie Jacksons; “Well,” she says, “here there's maybe one in three and that's pretty 
good”; (3) would do a whole lot of things— something that lets a kid excel in things other 
than books. Becky has heard that teachers are doing joint planning now and seem to value 
it. 

When the visitor ask what good teachers are like, Becky responds, “That's a hard one.” In 
her view, good teachers (1) do not have a kid “pegged in advance, you know, this is a bad 
kid because his parent is so-and-so”; (2) teach using “activity work”; and (3) foster 
communication between kids, parents, and self. The evaluator is struck by the first 
observation, which is an inevitable challenge for local people working in schools in 
communities where they grew up. Becky’s remark, for this reason, can be understood as 
generous. Not everyone will articulate this problem for a stranger. Asked if there are any 
mean or harmful teachers at the school, Becky says, “No.” She adds, “this community 
would not tolerate it; the person would be rode out of town.” Nonetheless, she believes 
there are a few people working as teachers “who are there just to collect paychecks.” She 
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seems to recognize an intermediate position between definitely harmful and partly 
effective. 

Becky reports that the school has tried a variety of grouping strategies in the past. She 
says they tried total departmentalization, down into the primary grades 2 years ago. “I,” 
she says, “don't have a problem with that, it worked pretty good for my kids.” Classes she 
says, were also broken down by aptitude by subject, with higher, middle, and lower groups 
reorganized by subject (a practice different from rigid “tracking”). 

Becky, a community insider, believes “the community” is far more closeknit than Abe 
Franklin suggested. “It's true,” she says, “that not everyone in the district personally know 
one another, but,” she continues, “as a local I know lots of folks, and I can usually place 
just about anyone.” She means “place” in memory, but probably also meaning that she 
can them in the web of coimections that constitutes “the community.” 

With this discussion, the topic of school consolidation is raised by the evaluator. Becky 
says she is proud of this school, its new technology, the teachers' being altogether in one 
location, but, she says, “I sometimes miss the small school” and the coimection with local 
communities. Although she identifies a tight community centered on the school, this 
comment indicates that she can also distinguish the four or five separately named places 
within the boundaries of the Ambler Hill district— places that once maintained separate 
schools. 

“But as far as the education, this is the best,” says Becky. Becky says that the Ambler 
Hill K-8 school is one school, “but the separation [in different wings of the building] is 
great.” Students can be easily brought together for events, as when the little kids get to 
view the big kids' stained glass projects. Becky says there was a planning committee, 
including the superintendent at the time, school board members, and also “community 
supporters.” She recalls that three folks were really influential, but doesn't name them. 
“The first two-three years were quite an adjustment [original emphasis]” she says. “For 
students and teachers and parents alike,” she adds. 

The little villages, didn't really lose a whole lot, she says, in the closings of their schools. 
She seems to sigh, and she pauses to collect her thoughts. “They sometimes used to shut 
down the whole school [that is, in the small schools before consolidation] to play ball, or 
picnic, or enjoy ice cream” (which once, she indicates, included making the ice cream). 
“We can't do that anymore. We can still do things for kids— arrange parties and that sort of 
thing.” “But,” observes the evaluator, “the whole school wouldn't cancel instruction 
because one fifth grade class is having a party.” Becky responds, “No that's not possible 
any more.” Indeed, under the prevailing wisdom, such an event would most often be 
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regarded as educationally irresponsible. 



Possibly because she realizes this, Becky’s reflections seem to raise the issue of school 
and district performance in her mind, and at this juncture she expresses concern about the 
effects of the state accountability system. What concerns her most about the Proficiency 
Tests is the perceived threat of further consolidation. “Will they,” she worries, “shut us 
down and put us on buses and ship us elsewhere [emphasis added]?” She really does use 
this phrase, including the significant substitution of the word “us” for “our children.” The 
part (the kids) stands for the whole (“us”— our shared identity, our common good, our 
common future). 

Becky reports that she has discussed mathematics and science during parent-teacher 
conferences, “but as far as curriculum,” she says, “no.” She indicated she had gotten 
involved with curriculiun issues surrounding a literature book that was suggested, but 
which she thought was inappropriate for 5th grade students. “There was material in it 
about a rape,” she said, and she felt this was inappropriate material for 12-year-olds. The 
book was removed [apparently from the list of suggested readings], “but some folks,” she 
continued, “used the occasion to inflate the issue beyond its significance.” 

Asked if she would like to add anything else to her remarks, Becky says she would like to 
see the Proficiency Tests discontinued and appreciate instead “what happens to these rural 
kids when they grow up.” 



Conclusion . The Project's situation in the AHSD seems fortunate to the evaluator. The 
district leadership and working faculty exhibit mutual respect, a desire for collaboration, and, in 
more than a few cases, an unusual measure of reflectiveness and candor (even with a stranger). 
The simultaneous existence of candor and mutual respect would seem to indicate a good 
foundation for the further development of a collaborative spirit, at least in the elementary and 
middle school of the AHSD. Collaboration for a common work, it should be noted, is the 
constructivist equivalent of hands-on and cooperative learning among the adults who inhabit 
schools. One might infer that the Ambler Hill teachers are trying to “walk their talk.”^^ 



Denzil Kline's critique of the Project summer 1999 sessions could be seen as evidence to support this 
inference. 
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So far as can be judged from conversations with the three community members 
interviewed, a diversity of views of education exists among parents. Evident among these three 
interviewees, is a remarkably strong concern for rural life. One interviewee suspects such a life is 
not exactly sustainable, but perhaps a somewhat transformed version of it will remain sustainable. 
Certainly, even the skeptic among these three representative of the Ambler Hill community is 
personally invested in the outcome. One might conclude from these stories that community 
involvement in curriculum is weak at best, since each of these representatives has played a 
stronger than average role in the Ambler Hill schools. This situation is common across the 
nation, but the Math Lab stained glass auction would certainly constitute a step in the right 
direction. Because it is a social event in celebration of the accomplishments of the district’s 
children, as well, it joins community life and academic purpose. 



Second Case Study: Troubled Academic Performance in Ambler Valiev 

The evaluator's first on-site impressions of the Ambler Valley School District (AVSD) 
come from Judith Clark, the Project REAL team leader. Formerly a teacher, and now serving for 
the first time as the AVSD's School Improvement Coordinator, Judy is an experienced teacher. 

She has served the district for over 20 years. She is a National Board certified teacher, having 
recently completed the certification process. 

She appears to be a logical choice for her current. For instance, early in her career she was 
named “Teacher of the Year” in the county (i.e., she was selected for the honor from among 
candidates representing all the districts in the county). She has worked most recently in an 
innovative curriculum development project partially fimded by one of the largest employers in the 
region (where the parents of many AVSD children are employed). And she has worked as a 
science specialist at the elementary level, in part helping other teachers conduct more engaging 
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science lessons. Judy holds a masters degree in educational administration and a principal's 
certificate. She has been offered principalships in other districts but has declined the offers— 
which involve substantial salary benefits— because, she says, of her commitment to serve the 
AVSD.^^ 

Likely major threat to school improvement in Amber Valley . Some of what I hear from 
Judy on the first interview, however, points to an ongoing staffing dilemma in the AVSD with 
critical implications for the sustainability of school improvement efforts like Project REAL. 
According to Judy, near-by districts— districts within an easy drive of this one— offer salaries for 
mid-career teachers that are up to $10,000 greater than those offered at Ambler Valley. 

“The way this works,” says Judy, “is that we train the teachers during their first years as 
teachers, and then they leave for better money.” The district's dilemma, then, would consist in 
possessing the expertise to direct change (Judy's competence), but in simultaneously having to 
confront a continuous assault on the organizational capacity for change. Unless one believes that 
organizational capacity does not depend on a modicum of stability among staff, one must 
acknowledge the severity of this dilemma. The evaluator receives confirmation of this staffing 
problem in subsequent interviews conducted in the AVSD. 

In fact, during visits to other districts the evaluator actually meets or hears about some of 
the good teachers who have left the employment of Ambler Valley. One of these former 
employees, in fact, now works for another of the demonstration site districts. In her old job, she 
taught advanced high school math classes exclusively (calculus, trigonometry. Algebra II). In her 
new job, she teaches a much less selective group of students, but reported that she “loved” the 
change and her new mission. In other words, this apparently excellent teacher— who left AVSD 



Judy accepted such a position for the 2000-2001 academic year— the second year in a row it had been 
offered to her. She now (November 2000) serves as principal of a small school a short drive from her 
home within the borders of a neighboring district. 
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where she taught an elite group of students (commonly viewed as an “easy” or more rewarding 
assignment) is now helping very different (more challenging) students in another district. 
Interestingly, the students she was helping at Ambler Valley would not contribute to 
improvements in the district's proficiency test scores, because these are the very students already 
passing these minimum competency tests at high levels. In her new district, however, this 
seemingly very competent teacher would have the opportunity to help a group of students whose 
better learning, all else equal, would most certainly contribute to improved accountability ratings 
in teacher's new district. 

The long term tendency of such defections might be two-fold in Ambler Valley. First, it 
means that natural (or informal) leadership is threatened by instability. Informal leadership (i.e., 
leadership not based on position) is critical to organizational culture, if little appreciated in the 
conventional literature on school reform (see Senge 1990, and Wheatley, 1994, for practical 
treatments of “dynamic systems theory”). In more immediately practical terms relevant to the 
Project, however, such defections mean that even selecting a Project team could be a challenging 
task. 

In all likelihood Ambler Valley would find it difficult to hire teachers who are both 
experienced and excellent. This situation would leave them hiring promising, but untested, young 
teachers as their best option. Once tested and experienced, however, such younger teachers would 
be likely to leave, so that their experience will redound to the credit not of Ambler Valley, but of 
some neighboring district. 

A district that continues to function in this context could realize ill effects on 
administrative decision-making as a result of organizational incapacity. Hypothetically, for 
instance, leaders might make assignments not on the basis of what is most instrumental in raising 
Proficiency Test scores, but on the basis of what is most agreeable to the talented young teachers 




60 



71 



they are in danger of losing to districts with higher pay scales. This hypothetical case may 
actually pertain to the former AVSD employee described above, who had a more “elite” 
assignment in Ambler Valley than she currently does in her new location. 

Superintendent’s hypothesis . In addition to speaking with Judith Clark, the evaluator also 
met briefly with the Ambler Valley superintendent, Deborah Oshepksi, and the principal of the 
middle school grades, Jerry Sanderling. Deborah Oshepski served previously as elementary 
principal, and has worked for the district for nearly as long as Judy. 

Amber Valley’s critically low Proficiency Test scores were a major concern of this case 
study. When asked what she though might be reasons for comparatively low Proficiency Test 
scores in the district, however, Deborah said she wasn't sure. She explained to the evaluator that 
district superintendents have the authority to change Proficiency Test scores thought to be in error 
in the case of individual students and that they were investigating this situation on the day of my 
visit. An observer might infer from this conversation that Deborah viewed problems with test 
administration, scoring, and reporting as contributing importantly to the AVSD difficulties. 
Perhaps, however, the conversation was too brief to permit other observations, or perhaps rapport 
was simply not established between the superintendent and the evaluator. 

Reception of the Project in Ambler Valiev . Judith, Deborah, and Jerry all stated their 
unequivocal support for the Project during their first encounters with the evaluator. Jerry told the 
evaluator that the summer 1999 Project conference was among the best he had ever participated 
in. Judy was eagerly planning to take her Project Team members to the Eisenhower National 
Clearinghouse in Columbus, to examine innovative mathematics and science materials. The team 
subsequently made this trip and found the visit exciting and helpful, according to Judy and to 
team members who mentioned it spontaneously during subsequent interviews. 

Principal Jerry Sanderling's interest in the Project was matched, in the evaluator's opinion. 



by his stated skepticism of “educational fads.” As a seasoned educator, Jerry is old enough to be 
personally familiar (as is the evaluator) with the inquiry-learning materials and philosophies of the 
post-Sputnik era (ca. 1960-1975). Jerry exhibits an appreciation for both engaged learning and 
structured activities, and he is fond of calling his new staff members “young lions,” whom he 
predicts, will mature into excellent teachers in short order.^"* 

Given this experience of educational change, Jerry's stated support for the project is 
perhaps telling. The evaluator intuits the support as genuine and significant and also sensitive to 
the fluxes of educational change. Jerry notes that “the bottom line” is the engagement of kids, 
which, in his view, depends on the enthusiasm and (intellectual) liveliness of teachers. Jerry did 
not use exactly these words, but they give the flavor of the evaluator's impressions of the principal 
of the two buildings in which the Project is implemented in Ambler Valley. 

Judy, but not Jerry, seemed concerned that the Project was not designed to tell districts 
what to do or how to do it, aside from specifying that improvement of middle-level mathematics 
and science was the aim. She was especially pleased (as was the case in every demonstration site) 
with the commitment of the North Central Regional Educational Laboratory (NCREL) to be 
involved with the Project. She was also concerned that NCREL sought a three-to-five your 
commitment, but that funding for the Project for additional years beyond the first was uncertain 
(at the time of the interview in September 1999). Judy expressed appreciation that the district 



”In a March 2000 interview, Jerry subsequently confided, “I doubt if I will be able to hang on to them.” 
He means that, once experienced, other districts are apt to lure his “young lions” away with higher 
salaries. One of the “young lions,” co-leader of the REAL team, however, is from the neighboring small 
city, is married to another teacher who works at the school, has recently purchased a house locally, and is 
evidently planning a family (according to the evaluator's conversations with him). When the evaluator 
suggests to Jerry that this teacher might be less likely to leave for slightly more money, Jerry says, rather 
noncommitally, “You might be right.” The evaluator had the impression that Jerry thought the evaluator 
might more probably be wrong! This exchange tends to support the view of the staffing dilemma 
presented by Judy Clark and reinforces the viability of hypotheses about the impact of the staffing 
dilemma at Ambler Valley. 
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would be able to purchase materials and that this would be the focus of the Ambler Valley 
project-choosing, purchasing, and using irmovative curricula and instructional materials. 

On the basis of these interviews— and particularly in light of subsequent interviews and 
observations in Ambler Valley— the evaluator concludes that Project REAL has been well 
received in the district. In fact, in interviews conducted in March 2000, every teacher interviewed 
had some knowledge of the Project— even when they had no involvement with Project activities. 

Decision-making in the Ambler Valley implementation of the Project . The evaluator has 
gathered some information about the way in which Project decisions are made in Ambler Valley. 
The Project has left practically all the choices in the hands of local leaders, and how the Team 
will decide these choices seems a critical focus of interest, since the options range from anarchy 
(or, more positively: organized chaos) to extremely hierarchical. 

Judith Clark stated unequivocally, “I want teacher input and participation, but the central 
office will be making the decisions.” The remark would seem to indicate an approach that is 
comparatively hierarchical. Participation is limited to allowing teachers “a hearing,” but power 
and authority are to be retained securely by administrators.^^ 

In actual practice, the evaluator believes some teachers exercise a modest degree of 
informal authority and power. The co-leader of the team is the upper-grade math teacher. Jack 
Plummer, a second-year teacher with an almost uncanny administrative sensibility for one so 
inexperienced in the profession— and Jack expressed to the evaluator his interest in pursuing a 
principal's certificate. (The interview with Jack will be summarized later in this report). Jack and 
Judy Clark, however, are the only representatives of the AVSD to attend the monthly Project 

In Western Ridges, central office members of the Project team took the opposite position. According to 
them, the district had too often attempted to enforce new efforts from the top. “This time,” they told the 
evaluator, “the teachers are in charge.” Respected teacher- leaders comprised the Project team in Western 
Ridges. In Western Ridges, also, the evaluator did not hear of “defections” to other districts for better pay 
or working conditions. 



REAL meetings. Jerry Sanderling, queried by the evaluator, seems aware of Jack Plummer's 
possible promise as an administrator. Given a hierarchical decision-making model open to 
teacher input, Jack's savvy outlook predisposes him to serve as co-leader, a predisposition that 
does not threaten the seeming hierarchical structure of decision-making in this team.^^ 

First-year implementation in Ambler Valiev . Data about ongoing implementation of the 
Project were gathered principally during a full day of classroom observation (classrooms of seven 
math and science teachers in grades 5-8) and from a subsequent day of interviews with four of 
those seven teachers. Teachers learned of the evaluator's visit on the day of the visits, not in 
advance; lesson plans were not, therefore, made in anticipation of the observations. Three of the 
seven teachers observed and two of those interviewed are not members of the Project team. 

Unplanned classroom observation provides the evaluation effort with the most accurate 
impression of day-to-day, business-as-usual instructional practice in the case study schools. The 
evaluator's inferences about the influence of the Project on observed events, however, is tenuous 
at best. In the interviews, by contrast, teachers were forthright in their claims about the influence 
of the Project on their pedagogical thinking and instructional practice. Changes in these two 
realms— pedagogical thinking and instructional practice— are critically related to the more formal 
goals of the project (which, of course, are not being evaluated in the current effort). 

These are the lessons observed: 

(1) In Jack Plummer's seventh-grade mathematics class, 18 students spend most of the 
period working on a complex rate problem; they work in small groups to generate 
and plot data; they use calculators; a sense of academic purpose pervades the 
classroom. The room is lively and active but does not appear chaotic. The teacher's 



The evaluator is not implying that participative modes of decision-making are naturally or universally 
superior to hierarchical modes. The term is intended descriptively, and one might substitute a phrase like 
“administrative-centered” rather than “teacher-centered.” In particular, if staffing is unstable, as it 
apparently is in Ambler Valley, there are good reasons for administrators to retain decision-making 
authority and power. 
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control is more tacit than explicit. 

(2) In Felicity Smith's eighth-grade science class, 14 students play a science game firmly 
directed by the teacher. The object of the game is to remember accurately facts from 
the previous night's assigned reading. The game takes most of the period and 
consists, surprisingly, of just five questions. Students appear remarkably contained 
in comparison to most of the other classrooms observed this day, and the teacher's 
control seems very explicit, even somehow intimidating. 

(3) In Gillian McMann's science class, about 22 students are working, according to 
Gillian, on a review lesson for the Proficiency Tests, soon to be administered. The 
concepts considered are presented in a workbook titled Sharpen Up on Ohio Science. 
Part of the lesson involves students working (a) individually and (b) in groups of 3 or 
4 to complete the workbook questions. The other part consists of a whole-group 
discussion of responses from the small groups. The discourse centers on making 
connections and deriving inferences. 

(4) In Ben Lively's fifth-grade science class, about 24 students are beginning a new 10- 
day project (growing bean embryos under varied conditions). The classroom is noisy 
and active, and filled with science artifacts. The classroom atmosphere is 
simultaneously relaxed (students talking sociably with one another) and busy (groups 
of 2 students working collaboratively and intently). Compared to other classes 
observed this day, there is comparatively little shared discourse. 

(5) In Shelba Roberts' sixth-grade mathematics class, about 22 students work on 
evaluating simple algebraic expressions. Students work individually throughout the 
lesson, which alternates between students' performing the assigned work and sections 
of debriefing or instruction. The class is very quiet and orderly, and the academic 
focus of instruction is quite clear to a visitor. Students seem engaged. 

(6) Kayla Dickinson, like Gillian McMann, was conducting a lesson intended as a 
review for the upcoming Proficiency Testing. The fifth grade had been grouped for 
purposes of “intervention” according to how many Proficiency Tests a student had 
passed. This was, it turns out, the group that had passed all tests. Today they were 
taking an off-level practice test. The decision to deal with this group in this way 
struck the evaluator as curious at best.^’ 



This judgement of the evaluator’s perhaps deserves additional comment. The intervention block had 
been created to help address “academic deficiencies” among other students. These students, however, had 
verified the adequacy of their test-taking skills, and so, to drill them on such skills seemed an inefficient 
or even counterproductive use of their time. The caveat to this observation, of course, is that the evaluator 
saw the class just once, and generalizations based on a single observation are (at best!) dubious. 
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(7) Samantha Frisk's eighth-grade science class has the assignment of defining 16 

vocabulary words and using each in a sentence. The class completes this exercise in 
the computer lab, using either AppleWorks or ClarissaWorks, it seems. 
Approximately half the class is touch-typing; the others complete the assignment 
more slowly, hunting and pecking. This classroom will turn out to be the only class 
observed, in either case-study site, in which computers are used in any way. 



The observations suggest that there is indeed a foundation in practice on which the Project 
might help teachers to develop instructional approaches in the ways it aims to promote. The 
Ambler Valley classroom observations reveal a remarkable variety of pedagogical and 
management sensibilities. The evaluator is refraining from offering much critique of the lessons, 
except to observe that elements of a constructivist-or hands-on-approach to learning were evident 
in most of these classrooms. Ben Lively's classroom was even a classic instance of the approach 
(and he is justly well-known in the district for his passionate commitment to teaching in this way). 

During the subsequent interviews, four of these teachers (i.e.. Jack Plummer, Ben Lively, 
Shelba Roberts, and Samantha Prisk) elaborated their pedagogical thinking and their perspectives 
on the Project at considerable length. The guiding questions appear in Appendix D. 

The evaluator, in each case, asked the teachers what was the best and worst they could say 
about the project (following Stake's 1975 counsel to do so), but also asked questions about 
barriers to or facilitators of the sorts of change intended by the Project, about interviewee's 
notions about teaching and about, in particular, their own teaching, and even more particularly, 
questions about changes in their own teaching. 

The teachers at Ambler Valley offered the following remarks in response to the evaluator's 
query about the best they could say about the Project: 

Team member : Within 2 or 3 years we should have the resources to do hands-on science. 

It will be good for the kids [because they] have not yet been educated to learn this way. 

Team member : The Project has gotten us to work with each other more— we did this less 



last year. Lots of teachers are nervous but excited; the Project has provided evidence that 
something works. A lot of teachers are accepting the Project; resistance is mostly among 
the older teachers. 

Teacher who is not a team member : Project REAL has definitely encouraged teachers to 
generate more hands-on activities [this year]. 

Teacher who is not a team member : I don't honestly know much about the Project. 

They're [sic] getting materials and summer training is being offered. [Adds: “I'm not able 
to participate in the summer training events.”] 

In response to the evaluator's query about the worst they could say, the teachers (same 
order as above) replied: 

Team member : I find it all kind of confusing, but the feeling of confusion might be just be 
the result of being a [yoimg] teacher. 

Team member : The monthly meetings are not productive enough; but this view may just 
be lack of patience on my part. I would like to see everyone involved in the 
communications loop. 

Non-member : I'm not officially informed about purposes of Project REAL. [This is an 
older teacher who attends all local team meetings despite not formally being a team 
member, and who is regarded as an active proponent of hands-on instruction.] 

Non-member : I don't really know, [pause] Teachers who are involved say it takes up too 
much time. Best thing may be that they get laptop computers; but one has said they can't 
find time to use it. 



Responses to these questions fi-om these teachers in Ambler Valley indicate overall 
interest and support for the Project, but the evaluator senses they also indicate moderate 
misgivings. The team remembers' remarks about “impatience” and “confusion” are possible 
evidence of such misgivings. Neither of these team members has taught longer than three years. 
The reason they are team members, in fact, as it turns out, is due to the departure of two teachers 
who had been named to the team when Project REAL applications were initially submitted at the 
close of the 1998-1999 school year. Between that time and the start of the new school year, 
however, there was a turnover in 2 of the 4 teacher-members of the Ambler Valley team. Observe 
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that this event tends, once more, to corroborate previous hypotheses about the staffing dilenuna in 
the AVSD. 

Teachers interviewed had a range of observations about barriers to and facilitators of 
change; in fact, however, most of their comments concerned barriers and relatively few concerned 
facilitators. One of the interviewees also reported that discussion of obstacles to Project 
implementation “hasn't surfaced” at AVSD team meetings. According to teacher interviewees, 
however, the barriers include the following: 

• exclusion of teachers who are less receptive to Project aims; 

• older teachers (i.e., a reason the opposite of the first item); 

• isolation (need for mentors, networking with teachers in other districts); 

• teachers' personal capacities (implied by nearly all interviewees); 

• district leadership; 

• team membership; 

• misalignment of curriculum and instruction; and 

• lack of materials. 

Ambler Valley interviewees mentioned or implied the presence of two facilitators: 

• Project REAL and 

• one's own capacities. 

Conclusions . The evaluator concludes that Project REAL is helping individual teachers in 
Ambler Valley more clearly consider the option of using hands-on activities as one way to ensure 
better task conunitment from students. The evaluator does not, however, sense that the Ambler 
Valley organization is successfully engaging the Project for wider change. In interactions with the 
evaluator, teachers tend to narrate their own practice and to focus on their own dilenunas, rather 
than on making connections among themselves, or between themselves and organizational 



purposes. The contrast with other demonstration sites is marked in this regard. 

The district's reportedly weak ability to either attract experienced teachers or to retain 
excellent young teachers could be a strong influence on this feature of organizational culture, but 
it is clearly a contributing factor in the operation and implementation of the Project. Difficulties 
with the continuity of staffing-and possibly other or related features of organizational culture— 
seem to the evaluator to have the effect of concentrating care for organizational continuity and 
purpose in the hands of the district administration. Except for the older teachers (who are 
generally characterized as a barrier to change), only the administrative leadership appears to have 
the organizational memory necessary to foster and direct change in this organization. 

The result of this conundrum— as it seems to the evaluator— is a comparative feeling of 
isolation and powerlessness among teachers. They report feeling isolated from one another; from 
the administration; and even (though the evaluator finds this testimony difficult to accept at face 
value) from the purposes of the Project. At the same time. Project REAL, according to the 
teachers' own testimony, is understood as requiring more collaboration and cross-fertilization. 

The expectations of the project— which seem well understood in this regard— may thus have the 
unintended effect of reinforcing the teachers' apparent powerlessness in this district. Indeed, any 
improvement project would seem to run such a risk in this district. 

In summary, the Project seems well positioned to help some Ambler Valley teachers 
improve instruction within their classrooms; it seems less well positioned to help foster an 
organizational culture that will propagate these changes more widely or sustain the changes that 
accrue as a result of Project successes.^* This judgment is a prediction, of course, with an 



A recent study of the Alaska Rural Systemic Reform Initiative, conducted by the Northwest Regional 
Educational Laboratory and the University of Alaska-Fairbanks (October, 1999) reported that “sustaining 
reform, though difficult, was not an impossible task in rural Alaska. The most significant barrier to 
sustaining reforms was persistent [staff] turnover.” Some districts in rural Appalachian Ohio may be 
facing the same principal barrier to reform that confronts districts in rural Alaska. 
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understandably wide margin of error. The evaluator is looking for a serviceable measure of 
organizational capacity that might provide additional data as well as provide additional insights 
about these tentative conclusions. 

Report of Additional Site Visits 

Interviews with local Project team members and classroom observations were conducted 
in Western Ridges and in Upper Ridges districts. The visits served principally to gather 
contrasting data to assist in interpretation of the case study work and, most particularly, the cross- 
site teacher survey, reported in a subsequent section of this report. For these reasons, the 
synthesis of information gathered during these visits will not be reported as elaborately as was 
information from the case study field work. 

Western Ridges 

Molly Shamblen, a longtime math teacher and local team member in Western Ridges, 
reported that the Project helped build more cross-school unity than had been evident previously. 
Recently the local project has started a monthly Project REAL newsletter. When asked to 
nominate a most changed teacher, Molly nominated two teachers in the “Southern Area” middle 
school. One of these teachers in that remote part of the county district is offering algebra to 
eighth grade for high school credit, she observes. Molly says the Project needs more time to have 
real influence. Mathematics and science teachers are no using their personal planning time for 
Project REAL activities, and she believes that teachers need to have more time for professional 
development built into their schedules. (The evaluator had previously observed Molly conducting 
a TIMSS workshop for teachers in her school, and concludes that she has absorbed the Study’s 



observations about the conditions of teachers’ worklives in other nations.) 

As for feedback to Project staff, Molly wants help with ways to “get everyone to 
participate.” She believes, as well, that the Project team needs to continue to receive professional 
development for another 3 to 5 years. She also notes a need for additional software and training, 
which she regards as a weak point in the district. She especially appreciates the attention to rural 
schools provided by the Project, which, she says is long overdue. Molly observes that, in Western 
Ridges, an equal number of younger and older teachers is involved in the Project, and that 
involvement here is “not a question of age but of who's frustrated and therefore motivated 
[emphasis added].” By this remark, Molly seems to indicate that she regards challenges and 
perceived shortcomings (“frustrations”) as essential to growth. This is a complex appreciation of 
the interaction of working conditions and commitment. 

The evaluator is accompanied to schools for observations by Amanda Billings, a central 
office administrator and local Project team member. Conversation is continuous. Amanda says 
that improving achievement test scores has been a primary concern in Western Ridges for 
decades; accountability in this sense is nothing new. The fearful new part, she says, is the threat 
of state takeover. “That's the essence of accountability: If you are too poor and lazy to do it, we'll 
do it for you,” she says. Project REAL, says Amanda, is the first time the district has looked at 
what they need and given it necessary commitment, not just adopted something and “rammed it 
down everyone's throats. The big Project REAL people are teachers [original emphasis] and so 
reception is better than if the program relied on mandates from the Central Office. Our team 
leaders have been recognized by peers; they are already leaders [original emphasis],” and she 
cites Molly Shamblen as an example. 

Amanda reports that teachers have left Western Ridges to visit math and science programs 
other districts and have received more professional development than they ever had in her 20 



years with the district. The district is adopting new curricula (NSF-approved), and introducing it 
slowly. The district, she says is used to Saxon, but awareness of other options “is on the rise: 
math teachers,” she observes, “are tough.” She appears to mean math teachers do not change 
pedagogical models easily, but that the Project is making headway in Western Ridges. 

Observations include a range of teaching from very much “hands-on”— an eighth grade 
science class in which students are working in teams to gather, record, and inspect data— to a 
much more traditional textbook based science lesson centered on teacher-talk. 

The eighth-grade science teacher, Cynthia Owsley, in the interview with the evaluator, 
credited her application for National Board Certification with significantly altering her approach 
to teaching. “I teach this way,” she said when asked, “because I’ve analyzed my understanding of 
science....! need to that to see the big picture.” 

Cynthia says, “I don't follow the textbook anymore, and I do many more labs.” She has, 
she says, learned how to design labs with a specific teaching lesson in mind. Nonetheless, she 
says, “I’m still very much in middle.” She is conscious of maintaining a balance between 
explanatory and inquiry-based learning. The evaluator gets the impression that, though conscious 
of so doing, she remains uncertain of the extent to which such balance is pedagogically necessary. 

Cynthia is not on the local Project team, however, and is not able to give much detail 
about Project activities. She clearly seems, nonetheless, to be working in the spirit of the Project, 
and her collaboration with it is doubtless unwitting and freely given. She reports, for instance, 
providing copies of materials relevant to national standards and current thinking about “the big 
picture,” but, she says “I’m not sure they've used them; I rather doubt it.” At the same time, this 
teacher insists that slow change is going on, “even among more traditional teachers.” 

When asked her view of “hands-on learning,” Cynthia says, “Hands-on helps, but it will 
not solve the problem [of low achievement]. There are,” she continues, “large groups of apathetic 
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or disconnected kids who know science but will not perform for a grade!” This is an 
extraordinarily astute observation about the disjunction between school cultiare and local culture, 
so far as the evaluator can judge. A large literature on the topic exists in urban education and in 
American Indian education, but comparatively little in rural education (Eliot Wigginton’s 
Sometimes a Shining Moment is one of very few such accounts). Cynthia is the only teacher 
interviewed during the course of the entire evaluation to raise the issue of improvements in 
achievement in this way: as an issue of cultural disjunction and not of the insufficiency of local 
culture or the poverty of local aspirations. 



Upper Ridges 

The Upper Ridges Project team is clearly led by Jason Poston, a high school teacher who 
generally teaches upper-level academic courses and is known to everyone by the nickname “doc.” 
When the evaluator asked how Jason received this nickname, the administrative representative on 
the Project team, Sally Jones, merely shrugged.^^ The reason seems evident, however. Jason 
projects a thoughtful, good humored, and pleasantly self-effacing introspection. “Doc” seems to 
suit his temperament and academic demeanor. A play of perceived irony continually crosses his 
face, and, in fact, is evident in his discourse. 

Jason judges that the Project has “been a real good thing for the district, with spill-over 
into other curriculum areas.” The initial curriculum audits, he says, helped the district frame its 
response (in its Continuous Improvement Plan) to the state’s placement of Upper Ridges on 
academic emergency. The Project has enabled collaborative planning times in Upper Ridges and 



Sally, who works in the central office located in the new Upper Ridges 7-12 high school building, says 
that Jason is the real team leader, and that her role is tangential. Because of the decision to study Ambler 
Hill and Ambler Valley districts intensively, the evaluator knows the participants in the other Project 
locations much less well than he’d have liked. A number of the teachers in this district, in particular, 
seem thoughtfully complex and academically centered. Jason is certainly among these teachers. 
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provided support for one day a month of professional development activities. As a result, says 
Jason, the decision has been made that all Upper Ridges students will take algebra in the future. 
Jason muses, “I used to be the only person who watched data; now everybody is and they can 
interpret it.” 

When asked to nominate a “most changed” teacher, Jason refers to a fellow mathematics 
teacher (whom the evaluator was scheduled to observe but who was, unfortimately, giving a test 
on day of this visit). Jason observes, “It’s hard to get him to change after 28 years, but he has 
signed up for Connected Math training this summer.” (The irony here is that Jason has been 
teaching for 25 years.) Jason freely admits that his own teaching has changed considerably in 
recent years. Why? “It’s more /wn to teach this way,” he replies. 

Jason would like to see the Project implement more sharing between schools. 

“Networking is important,” he says. “We got to see everyone in Chicago [that is, at the annual 
meeting of the National Council of Teachers of Mathematics, an activity funded in part by 
NCREL] and that was great.” Jason says he’s aware that the Project leadership at OU Southern is 
making plans to address this possibility. As team leader, Jason would also like to see financial 
reports “handled better: we're not sure what our [Project account] balance is.” 

The evaluator also interviews Sally Jones. Her remarks are very similar to Jason’s, but 
she adds that “the work of the local Project team (five teachers) is rubbing off on the rest.” Sally 
says she’s appreciated the Project’s support for professional development and believes that, as a 
result, “there are a lot of revelations taking place” among Upper Ridges faculty. 

Administratively, she, too, would like to see more timely Project account reports from the central 
Project leadership. She’s been very pleased, she adds, with NCREL’s contribution to the Project. 

The evaluator observes a number of classrooms in Upper Ridges, but has been most 
curious to observe and interview Rita Casto, the “defector” from Ambler Valley, mentioned 



previously. Rita says that the reason she teaches is “my love for math. I love to share my 
knowledge with others; I’m a very positive-thinking person and I do not like to hear students say 
1 cannot do this!.”’ When the evaluator asks about changes in her own classroom practice, Rita 
observes that she is “getting more experience and getting wiser.” In particular, she says, she is 
“getting used to the fact that class can be loud and focused at the same time; I'm 70% comfortable 
with this now, as long as they are talking math [original emphasis]. ” This change in my teaching, 
she reports, started during the last part of my final year in Ambler Valley.” She observes that “the 
grouping, the role of kids, ” pushed her in this direction. She seems to be saying that she came to 
object to the role of kids as winners and losers in schools. 

What she says subsequently seems to confirm this impression. Rita says what motivated 
her to change was when a former principal at Ambler Valley High School asked her “if she'd ever 
made a difference.” She replied, “sure, this kid or that kid went to the Ivy League.” He replied, to 
the contrary, that “no, they'd have gone in that direction without your help. No, I mean, have you 
ever changed some kid’s life in an important way?” That, she says, got her to thinking, and the 
remark helped her take this job. Rita, as well, was raised in the Upper Ridges district and 
graduated fi'om the Upper Ridges High School. Salaries are better here, as well, but the evaluator 
does not doubt Rita’s authenticity: her story has the ring of revelation and epiphany to it. 

One of the reasons the evaluator wants to speak with Rita, not surprisingly, is to discover 
her opinions about Ambler Valley. He asks bluntly, “What's the problem at Ambler Valley?” 

Rita answers instantly. “Basketball,” she says, “rules.” And they prioritize funding and concern 
that way, according to Rita. 

Rita also insists that the Project is responsible for the mathematics department’s decision 
to start teaching transitional math in the ninth grade next year. They are also, she reports, 
searching for better forms of assessment, and, in particular are “checking out portfolio 




75 



86 



assessment.” 

The class observed appears to be an eighth grade general math class, and the topic for 
today is the relationship between fractional and percentage forms of ratios. Rita begins by 
debriefing the complaints of a substitute teacher with the class. She makes it clear that talk about 
math is encouraged even if loud, but that she expects good behavior. This gentle chastisement is 
not over played, and it is done earnestly and quickly. The kids don’t balk or object. 

Rita asks the class what they were talking about last. They respond, “page 297!” This is 
not the response Rita is looking for. “No, no,” she says, “I mean the concept.'' “Percents” they 
say. Most of the class work is taken up with drill related to the focal concepts. Students work 
these drills, however, collaboratively, and Rita consciously and consistently cultivates active 
collaboration. She has a rule: “ask three before you ask me.” That is, the group must pool its 
thinking before consulting the teacher. Her emphasis is consistently conceptual, and she 
emphasizes that the task in hand is to “simplify, not reduce.” These, one must remember, are 
students whose learning is often debased pedagogically to rote processes. Clearly, Rita holds 
much higher expectations for these kids. 

Rita circulates among groups, one group at front not talking math, but adolescent personal 
concems-dates and so forth, and Rita simply moves on. The evaluator wonders what will 
happen, given Rita’s insistence that math be the focus of conversation. Eventually the lead kid 
says, “We gotta do math” and the group gets down to work. The “lead kid” is girl who, in the 
way of “mature” working-class 14 year olds, projects a tough image, possibly in the attempt to 
appear more jaded and adult than they really are.'”’ Rita spends considerable time throughout the 
lesson encouraging kids to think, record what they know, help each other, and give explanations 



This speculation is intended to describe the event and the moment, and is not offered as a completely 
accurate evocation of the student. 



to one another. 



The lesson draws the evaluator (a former math teacher) in, and he begins to interact with a 
team of two students sitting near his observation post. They are stymied by a compound fraction, 
which they are supposed to convert to percentage form. In due course, Rita centers her attention 
on this team of two. She sits down to talk not about the calculations but about why percents are 
practically important— usefulness in shopping, in this interchange. She shows how to calculate 
percentage quickly as fractions, for instance, a $6 savings on $24 purchase under a 25% discount 
arrangement. One of the students looks amused, incredulous,— the most animated expression this 
class. He is very clearly appreciative of the teacher's attention; this, it turns out, is a special 
education student. 

Near the end of class, one kid calls another “dumb” when says he's not ready for 
tomorrow's quiz. Rita makes the kid apologize, then re-runs, to the entire class, the perspective 
on the practicality of percentages that she’d just shared with two kids with whom the evaluator 
had interacted. 

The evaluator drives down the road to the elementary school, to do some field work 
among the lower middle grades. Upper Ridges now maintains just two buildings, located within 
about 6 or 7 miles of each other. The high school is new, attractive, air conditioned. The 
elementary school is blockish, with an “institutional” feel to the building, although the school 
climate feels fine. There is no air conditioning in the elementary school, and the classrooms are 
warm in May, but by no means stifling. Windows are open everywhere. 

The evaluator’s notes observe, this school building “almost has the feel of a third- world 
school— blocks of classrooms splayed out along long corridors.... the very design seems shockingly 
thoughtless; it seems too large and ugly, to boot.” Ambler Hill K-8 school is about the same size 
for grades K-6, but feels oddly smaller and decidedly more comfortable. 
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Emma is running a very inductive lesson with this class, and doing it well. The kids are 
engaged, even excited by the puzzle she has posed them, which is a logical puzzle. The kids work 
in team, debate the applicable logic in their groups, and then report to the class as a whole. They 
do this easily and well, with good focus the entire time. This was a mathematics class without 
numbers, and the prospect was pleasing to a stranger, so seldom does logic appear as an explicit 
issue in mathematics classrooms. 

During the subsequent interview, Emma reports that the most changed middle-level 
teacher “is similar to me, but breaking away from the textbook— she has moved further away from 
the textbook this year, according to rumor.” The evaluator asks Emma if she has actually 
observed her colleague’s teaching, in this case, and she responds that she has not. 

Emma says that she has recently begun to use more “hands-on” lessons (the evaluator 
would classify the observed lesson under that rubric). She wants, she says, to use “many routes to 
learning” and wishes she could do still more and “reach more kids.” 

Emma has been teaching 10 years. She has been involved with constructivist perspectives 
all along, she says, participating, for instance, in “Project Discovery 95.” When the district 
adopted Saxon math two years ago, she followed the Saxon regimen as an antidote to the old 
textbook (which the evaluator infers she did not like). Saxon was chosen, Emma insisted, on the 
recommendation of the Ohio Department of Education, and she was, she said, initially a big 
supporter of its use. Project REAL, she says, has helped her switch to a more hands-on approach 
and helped “moving her off the strict Saxon” regimen. Like many teachers, Emma seems 
concerned to strike a balance between inquiry learning and more discursive, teacher-directed 
instruction. She seems acutely aware of the tension between the Saxon principles and 
“constructivist” principles. 

Emma would like to see the Project include more of those who “don't participate” on the 
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local project team and in training. Emma believes that the Project has demonstrated its staying 
power this year. She observes, “The fact that they really have been here all year and proven their 
dedication--thafs the best.” The implication is that most school improvement efforts in her 10- 
year experience were brief, superficial, and ephemeral. 



Emergent Themes and Issues 



For the purposes of this report, the evaluator distinguishes “themes” from “issues” as 
follows: “themes” are recurrent cross-site topics of discourse relevant to the Project or its aims or 
methods; “issues” are unsettled questions or dilemmas that the Project confronts. Themes, then, 
are commonalities of what is said by participants about the Project (or its aims or methods), 
whereas “issues” are inferences that the evaluator draws from experience, data, analysis, and 
reflection (including reflection about evident themes). The themes and issues are “emergent” in 
two senses. First, they “emerge” from the evaluator’s reading of the field notes. Second, they 
“emerge” from the ongoing reality of Project sites, and, thus, are necessarily not complete. They 
are, in other words, continuing to emerge in the real world of the Project. 



Themes 

This report highlights five themes that are prominent in the discourse of participants to 
this point (and in no particular order of priority). The meaning of these themes is considered 
under the rubric of the three issues subsequently identified (that is, following the presentation of 
themes). Readers of this report, then, are cautioned that the “themes” are offered more as a 
representation of facts than as an interpretation of facts. The “issues” sections presents the related 
interpretations. The five emergent themes are as follows: 
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1. negotiating Proficiency Tests (several sub-themes: accoimtability procedures, fear of 
sanctions and “takeovers” by the state, revising curricula) 

2. realizing constructivist pedagogy and curricula (many sub-themes: hands-on 
learning, engaged learning, thinking-skills, problem-solving, meaning, wholism); 

3. engaging professional development (several sub-themes: reflection, in-service 
opportunities, collaboration, isolation); 

4. experiencing the ambiguities of change (several sub-themes: role ambiguity, 
communication within site, communication across sites, “politics”'*', concepts and 
purposes of change); and 

5. realizing collaboration and autonomy (this theme is intended neither as a dualism nor 
as a continuum, but as a complexity, and it verges on being “an issue”)/^ 

Negotiating Proficiency Tests and accountability . Proficiency Testing and accoimtability 
are mentioned frequently by interviewees in the field notes. A range of views exists about the 
testing itself, but all team leaders characterize an influence that they believe to be more positive 
than negative. These educators view the existence of the tests and the accountability judgements 
of the state as a lever for improving curriculum and instruction at the local level. 

At the same time, resentment and fear of the unknown are common, if not prominent 
themes. When asked what they are afraid of, interviewees said something like “being taken over 
by the state!” Being judged as in a state of “academic emergency” or “academic watch” is 
evidently taken as a serious condition (akin to being on hospital “critical” or “serious condition” 



The term is used loosely here to indicate a realm of action often perceived as unsympathetic (at best), 
more often irrational, and desperate venal at worst. It refers to both state- and local-level action and 
leadership. This is the usual sense in which Americans employ the word (i.e., as a pejorative). 

For example, under the conditions of complexity, a teacher might be rated high on both qualities 
simultaneously or low on both qualities simultaneously. 



lists) by most interviewees. Community members interviewed in Ambler Hill were especially 
concerned, and not particularly appreciative of accountability. 

Realizing constructivist pedagogy and curricula . While Proficiency testing and 
accountability constitutes a very common theme, it would not be correct to infer that it is the most 
pressing or salient concern of classroom teachers. These participants are quite concerned and they 
do— in all districts— report spending considerable time “reviewing” material to prepare students to 
perform well on tests. 

However, the concern to improve instruction (as contrasted with the marginal upward 
movement of test scores) seems to runs deep among a majority of teachers interviewed. Many 
were engaged in a struggle that the evaluator would call “truly professional” to improve their 
teaching and sharpen the learning of students. Sometimes, it seemed to the evaluator, this was an 
intense and lonely effort. Sometimes, it was less lonely, and, sometimes, less intense. But the 
effort seemed ongoing nearly everywhere the evaluator visited, and most improvement envisioned 
by teachers was clearly along the lines of “hands-on” and “engaged” learning, that is, clearly in 
the constructivist mode. Historically, this is a sea-change in the conventional wisdom of school 
practice. Not so long ago, behaviorism ruled the educational roost. 

The evaluation report will not engage in any formal analyses of the curriculum audits 
prepared for the demonstration sites. The evaluator has, however, read most of these audits and 
their recommendations and interviewed leaders in all five demonstration sites about their 
perspectives on the validity and usefulness of the audits. Most of the audits, in fact, rate schools' 
curriculum and instruction about average overall, vsdth considerable variation in ratings of 
individually measured practices. Team leaders all indicated the audits seemed to them to possess 
“face validity” (though not, of course, in so many words). Like Proficiency tests, they were 
viewed as a lever to edge change forward. Unlike Proficiency tests, however, they were not 



regarded with fear. 

It is not apparent, however, that the teams refer continuously to the audits as they do to the 
tests. The audits may have served a primary purpose of legitimating the constructivist agenda and 
focusing discourse and thinking about change along those lines (which are, as noted at the outset 
of this report, the major tenets of national reform efforts in mathematics and science). Case-study 
site teachers are not only using this discourse, they are reportedly changing their instruction 
accordingly, sometimes with considerable evident success, so far as can be judged by 
unarmounced classroom observations and interviews among faculty. 

Engaging professional development . The previous theme, in the evaluator's view, 
concerns the personal-professional talk and thinking of individual teachers engaging in the kinds 
of change the Project seeks to enable. This theme indicates a more abstract realm of change, that 
is, a more self-conscious arena of rationalized and intentional change. In this mode, teachers and 
faculties engage in the professional project of imagining themselves into the evolving experience 
of their careers. 

What will I (or what will we) be like in a year, in three years, or in five years? What does 
it mean? What do I (or we) want? How do I (or we) get there? What are my (or our) 
alternatives? Do I want to take a formal leadership role? To this last question, several evident 
teacher leaders in the Project said, in effect, “Yes, we are thinking about administration or 
supervision.” The evaluator was most moved, in this work, to see the reflection sub-theme 
emerge in conversation with teachers at both Ambler Hill and Ambler Valley. 

Denzil Kline's reflections on control in the classroom are actually poignant. In his 
classroom, he seemed fully in intellectual as well as behavioral control to the evaluator. At the 
same time, he claimed (a) to “be learning along with the kids” and (b) to tolerate challenges to his 
knowledge (from students), using them to model his own curiosity and incomplete knowledge as 



occasions for continued inquiry and mutual learning. Denzil told the evaluator that he had, with 
considerable effort, learned not to feel threatened by these encounters. Denzil’ s response was 
quite different from the one given by a teacher at another site, who felt such challenges inevitably 
undermined his classroom presence. Denzil’ s case clearly shows (as intimated by Western 
Ridge’s Molly Shamblen) frustration can easily become a prompt to professional growth. The 
teacher who felt threatened by intellectual challenges from students could be on the threshold of 
growth. 

Quite a few of the Project teachers interviewed are telling themselves complex, subtle, and 
thoughtful stories about their own teaching and the learning of their students. It's an honor and 
delight for an evaluator to witness this degree of every-day reflectiveness. The Project inevitably 
reinforces this habit, in the evaluator's view. To affirm this statement objectively would probably 
not be possible, but the evident quality of the thinking and the attendant internal discourse 
concern the same complex matters and views that the Project affirms. A concordance exists, at a 
minimum, and the field notes, as well as the synthesis of these five themes, attest to the fact. 

Experiencing the ambiguities of change . The uncertainty of the human condition means 
that change of any sort entails ambiguity. Not only is the future inherently uncertain, but planned 
changes set up a contrast between a projected or anticipated change that is inevitably at odds with 
the changes that eventually accrue. In other words, change rarely— if ever— proceeds as planned, 
and sometimes the best changes are realized unintentionally. Unfortunately, sometimes— and 
more often than anyone would like— good intentions produce counterproductive results as plans 
are implemented. 

Some teachers, especially those in Amber Valley, expressed “confusion” or “uncertainty” 
or just “ignorance” of Project purposes, intention, and operation. One very involved teacher even 
claimed to be “uninformed.” Actually, from the evaluator's perspective, this assertion fly in the 
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face of the teacher’s own classroom practice, which seemed extraordinarily responsive to the 
purposes and intentions of the project.'*^ Several interviewees claimed to lack the “big picture” of 
reform. In the evaluator's view, these expressions of concern and consternation are not criticisms 
of the Project, but more aptly reflect the assumption by teachers of an unaccustomed role. Their 
fhistration would ultimately be predicted to provoke professional growth (though for which of 
them, and to what extent is more obscure). 

This circumstance of teachers' worklives is ironic for the Project because the Project is not 
mandating change of any particular sort-even though it promotes a certain view of pedagogy (i.e., 
constructivism). Local educators (teachers and administrators) must decide the use of Project 
resources. They must decide on matters of inclusion and communication, and on matters of 
comparative emphasis (materials purchase or training, for instance). The circumstances of their 
worklives apparently leave some participants, or participants in some sites more than others, 
feeling unbalanced by the ambiguity of self-directed change. Few models exist for this sort of 
change, and the institution of mass public schooling has not actually been cultivated to permit 
such change to flourish.'’'' 

Realizing collaboration and autonomy . The essence of this theme is given by a statement 



This teacher had not been named to the Project team, because (as the evaluator heard from multiple 
sources) of Amber Valley had determined not to include teachers within a few years of retirement on the 
team. In fact, this “uninformed non-member” actually attended most Project team meetings, which were, 
indeed, open meetings. In claiming to be uninformed, the teacher is speaking with some irony. 

An interpretation seems in order, and it is provided in this footnote separately from the more objective 
discussion of the theme. In many way, states continue to supervise schools according to the factory 
model— with production orders issued from central management (i.e., the state-level, in terms of this 
analogy) and with standardized outcomes the desired goal. At the level of a teacher’s experience, this 
theme portrays the “bleeding edge” of standards-based reform, replete with dilemmas and contradictions. 
Some demonstration districts appear to be managing this dilemma and negotiating the related 
contradictions better than others. A relevant issue for the state would seem to concern the application of 
standards in ways that support diversity in place of standardization while maximizing instructional 
decency for students. 
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from one of the demonstration site leaders (and not, incidentally, from one of the case-study 
sites): “The only route to changing the instructional practices of less than wonderful teachers is 
with the leadership of wonderful teachers [paraphrased statement].” From interviews and 
conversation with teachers and administrators across all the demonstration sites, the evaluator 
concludes that this sentiment enjoys comparatively wide rhetorical support, but comparatively 
little active support in some demonstration sites. Western Ridges, the district with perhaps the 
biggest impediment (its large size and its varied communities, which preserve some of the 
traditions of the smaller districts consolidated about 1970), actually seems, after bad experiences 
with the alternative, to be implementing this approach. Upper Ridges also appears well along the 
path to implementing this view, as does Ambler Hill. In Ambler Valley, the intention apparently 
faces sharper threats to its realization. 

The influence of “wonderful” teachers is not just acknowledged, but evident in the 
statements of interviewees, to the effect that one teacher or another is an excellent teacher (from 
community members, for instance) or that one or two colleagues in particular have been very 
helpful (from fellow teachers) or that, yes, each level really does have a sort of natural or informal 
instructional leader (from administrators). The mystery is how these teachers thought to be so 
critical to improvement got “wonderful” in the first place. It was not, most interviewees seem to 
think, by design or by planned change. One instructional leader put it this way: “Good teachers 
help kids learn; they could probably teach without any materials-, we can't get enough good 
teachers, and we can't predict who will become a good teacher very well.” 

A few clues from the case-study interviews indicate that, instead of being mentored by 
wonderful teachers before them, these wonderful teachers have an encounter that somehow 
resonates with a pre-existing sense of mission, particularly the sense of a mission only partly 
fulfilled up to that point. The encounter might be an unpleasant professional event, or the pointed 
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question of a senior colleague (as with Rita Casto in Upper Ridges), or it might be the ideas 
explained in a new program (like Project REAL for Denzil Kline in Ambler Hill or like the 
receipt of hands-on materials, in the case of Ben Lively in Ambler Valley) or it might simply be 
the continuing evolution of a thoughtful and energetic disposition (like Jamie Jackson's Math Lab 
appeared to have been). 

These wonderful teachers, in fact, seem to achieve their degree of excellence through more 
or less autonomous action and reflection that helps them realize more fully their personal 
missions as teachers. They remain skeptical, skeptical of received wisdom, of the view of others, 
and, in fact, of their own experience. They accomplish this growth differently, partly by 
autonomous plan, partly by improvisation, and, it seems, never by mandate or imposition. Others 
often help them, but the instant of help, as reported to the evaluator, seems practically 
unpredictable.'’^ 

Moreover, some of these wonderful teachers exert a clear and extensive influence on their 
colleagues-an influence that is always indirect, respectful, subtle, and non-directive: “collegial,” 
in a word. Practically every site visited showed evidence of this network of possibly influential 
teachers. But these teachers encounter vastly different organizational circumstances, of course, 
depending on which district employs them. Thus, some enjoy circumstances that are, 
comparatively speaking, more conducive to collaboration of the sort that augments improvement 
efforts. Others, however, endure conditions more likely to fimstrate collaboration. 



Issues 

The evaluator identified the preceding themes through a comparatively lower-inference 



The assistance of others may depend in part on organizational culture, in part on organizational 
capacity, and in part of the moral and intellectual courage of individuals. 
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process of reflecting the statements of Project REAL participants themselves. Proficiency tests, 
constructivist pedagogy, professional development, the ambiguity of change, and collaboration- 
autonomy figure prominently, both explicitly and tacitly, in the discourse of the educators with 
whom the evaluator spoke. They represent quite evident major concerns related to the Project as 
represented in the lives of teachers and their administrative colleagues in demonstration sites. 

But these five themes also point to a number of critical issues for the Project as it 
continues to work toward improved mathematics and science instruction in rural Southeast Ohio. 
The evaluator characterizes these critical issues as follows: 

1 . organizational capacity, 

2. rural culture, and 

3. teachers' knowledge of educational change and reform. 

Like the themes, the three issues are inextricably intertwined with one another. They 
concern, of course, the organization (school and district); the local circumstance (rural); and an 
aspect of personnel (teachers' standing vis a vis reform). The issues are inferred by the evaluator 
from the experience of site visits, observations, interviews, and analysis of the field notes. 
Reference to these issues do not appear so explicitly in the field notes as reference to the themes. 
Issues are, however, tacit in the field notes, but the issues also “exist” at a higher level of 
abstraction than the issues, and, so it is not surprising that they should not figure explicitly in 
everyday discourse. 

But, more importantly, “issues” are not “themes” because they point not principally to 
topics of discourse, but to domains of complex underlying dilemmas. The issues, then, are 
inferred by the evaluator— and from the themes that seemingly concern the participants— as being 
particular dilemmas for Project REAL. 




87 



98 



Organizational capacity .'*^ The contextual and organizational differences noted in Tables 
1 and 2 seem to indicate quite substantial differences among the districts—differences perceptible 
without making any site visits at all! Visits to all demonstration sites, however, suggest to a 
visitor the existence of quite different senses of organizational ethos or culture, qualities related to 
the putative organizational capacity to educate students well.'’’ 

The intensive case-study visits, by contrast, provide deeper insight into organizational 
culture (if not precisely “organizational capacity”) than is possible from a single visit or the 
inspection of even vast amounts of demographic data. Though organizational contrasts in the 
case-study sites were perhaps suspected when the selection of sites was considered by the 
evaluator and Project staff, the issue of organizational capacity did not receive any explicit 
consideration during the selection process, according to statements from Project staff represented 
in the evaluator's field notes. The identification of this issue, then, seems to have been raised by 
the evaluator's interaction with the self-representations of participants (i.e., their self- 



The evaluator uses the term “capacity” instead of “culture” to indicate the organization's hypothetical 
potential to educate students well. “Organizational capacity,” in other words is understood as a construct 
that resembles an individual's “IQ” or “aptitude,” but that is instead applicable to an entire organization. 

It differs from “educational effectiveness” as individual aptitude differs from individual achievement. 
Though the Ohio Department of Education judges none of the demonstration sites as “effective,” they 
may nonetheless exhibit markedly different levels of organizational capacity. Please note that the 
evaluator is not distinguishing types of organizational capacity. In particular, “organizational capacity for 
change” is not entertained as a valid construct because the content of change is by no means generic, and 
one type of change may require organizational characteristics that contradict characteristics required by 
other sorts of change. Nonetheless, as a result of the identification of this issue, the evaluator now intends 
to include— as part of the planned cross-site survey— a brief measure of some feature of “organizational 
capacity” relevant to the Project themes and issues (most likely a measure of the organization's 
collaborative capacity). 

That is, a construct known as “organizational capacity” is alleged to influence school performance, but 
this influence is putative because empirical research on the influence is thin. Indeed, like most 
psychological or social constructs, this one has been measured in very different ways, with very different 
results. Overall, however, the connection to student achievement is very tenuous. The cross-site teacher 
survey (see Appendix C for the instrument), results from which are described in the next major section of 
the report, contains one efficient operationalization of the construct of “organizational capacity,” based on 
an extant instrument in the recent evaluation literature. 
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representations as “data” for the evaluation). In short, the stories told by interviewees clearly 
suggest to an informed observer that organizational capacity--and site-to-site difference in 
organizational capacity— is in play as this school reform effort. 

Ambler Hill and Ambler Valley, despite some differences (see Tables 1 and 2), also 
exhibit some similarities. In both districts, teacher salaries are reportedly low. Both districts, 
moreover, are located within reasonable commuting distances of metropolitan areas'**, and both 
enroll fewer students than the median Ohio district. The Project-relevant academic performance 
of the districts, as reported in Table 1, is also very similar— that is, substantially below expectancy 
levels set by the Ohio Department of Education. 

Contextually, however, the two districts are dramatically quite different. Ambler Hill 
would be said not simply to be “poor” but even to be “poverty-stricken” by conventional 
standards. Residents interviewed acknowledge this description, but they do not demonize the 
poor; instead, they speak of the dignity and “respectability” of people trying to retain away of life 
in face of poor odds. 

Ambler Valley would most likely be judged as affluent by conventional standards, with 
families with children enjoying mean family incomes almost two and one half times those that 
prevail among Ambler Hill families. It’s a striking and significant quantitative difference. 

Similar trends are evident in terms of adult educational attainment, free-and-reduced-price-meals 
rates, child poverty rates, and percentage of special education students served by the districts. 

In this light, though Ambler Hill is objectively much “poorer” than Ambler Valley, the 
small differences in overall achievement in favor of Ambler Hill seem very remarkable. They 



'** Ambler Hill’s nearby “declining industrial town” is located within an additional 20 minutes of a 
metropolitan county whose central city has a population just over 50,000 (for a total commuting time of 
about 50 minutes from the district center). Ambler Valley is located approximately the same distance 
from Columbus, which is, of course, a much more cosmopolitan metropolitan area. 



may, in fact, be absolutely remarkable, the judgement of the Ohio Department of Education 
notwithstanding. 

Hypothetical differences in organizational capacity characterize the two districts, so far as 
the evaluator can infer from interviews, informal conversations, and classroom observations. The 
sharp difference in staff turnover is clearly implicated in these hypothetical differences, as the 
case study reports indicated. Ambler Valley has consistently employed comparatively 
inexperienced teachers to replace more experienced departing staff members. 

These defections mean that other districts now enjoy the benefits of on-the-job 
professional development provided by the Ambler Valley district in past years to their (these other 
districts’) new employees. That expertise might be seen not as fleeting, but as remaining to 
benefit the state as a whole, but two reflections temper this thought. First, Ambler Valley has 
seemingly provided such development almost to its own detriment. Second, the benefits (of the 
defections to other districts) are governed by the marketplace and not necessarily by educational 
need.'*’ 

At the same time, older teachers are regarded with considerable suspicion in that district. 
The committed and energetic older teacher, perhaps the district’s “star” of classroom practice 
consonant with constructivist principles, visibly chafes under the offense of this prejudice. 
Prejudice against older teachers is notably absent in other demonstration sites, where older 
teachers who resemble their energetic colleague in Ambler Hill are acknowledged teacher-leaders. 

Ambler Valley can hardly re-allocate the resources that support on-the-job professional 
development, of course, because new teachers acquire this experience simply as part of their 



Rita Casto’s defection, however, certainly constitutes a counterexample of sorts, since she moved with 
her considerable talents from a wealthier district that paid poorly to a more impoverished district (her 
home district) that payed better. The position that the marketplace meets educational need best is held by 
some educational writers, but not by the evaluator. 



employment. In any case, a high rate of faculty turnover, coupled with the replacement of 
experienced young mid-career teachers by younger beginning teachers, and the comparative 
demonization of older teachers inevitably diminishes organizational capacity. It especially works 
to cripple organizational leadership, if one takes the enlightened view of “dynamic systems 
theory” that leadership in effective organizations is widely distributed (Senge, 1990; Wheatley, 
1994). 

Leadership from the superintendent, as well, seems to be regarded quite differently in the 
two districts. In Ambler Valley, many interviewees reported a more “top-down” administrative 
style, and a meaningfully higher level of “fhistration” with the central office leadership than was 
the case in Ambler Hill.^° It was suggested to the evaluator that the Amber Valley administration, 
for instance, through its own actions (unelaborated upon by informants) contributed substantially 
to the recent defeat by the public of the district's operating levy. The election result itself, in any 
case, is an indicated “vote of no confidence” in district leadership. Such votes are becoming 
increasingly common, of course, throughout the state, and especially in impoverished rural areas 
like those served by the Project. 

The defeat of the levy was quite discouraging to some of those exercising Project 
leadership in Ambler Valley. Interviewees, in fact, sometimes mentioned “low morale” as a 
general problem in the district, with the levy defeat seen as a further blow. Low morale is not an 
isolated condition, but one that reflects organizational health much more generally. Low morale, 
moreover, disables organizational capacity, and this is especially true in a human services 
institution like schooling. Teachers constitute the organization. Low morale among them is, in 
fact, an organizational energy sink. Psychological and physical energy otherwise available to the 



Once again, “frustration” is not necessarily a barrier to professional growth, but often serves to provoke 
it. 
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organization turns against itself, individually and organizationally, and the organization 
hypothetically fails to realize opportunities that come its way. Indeed, low morale is actively 
disorganizing (“energy” [Gk., erg] is the root morpheme in “organization”). 

A fourth inference that contributes to the evaluator's hypothesis of critical differences in 
organizational capacity concerns professional isolation. Professional isolation in Ambler Valley 
is inferred partly from the comparatively fewer references in teacher interviews to the influence of 
colleagues on one another in that district based on field note data for Ambler Hill, Western 
Ridges, and Upper Ridges. The construction “older teacher,” as well, indicates something 
different in Ambler Valley as compared to the other districts. In Ambler Valley, as previously 
noted, “older teacher” is often used to flag resistance to the sort of pedagogy valued by the 
Project. Some older teachers in Ambler Valley did seem to exhibit less receptivity to 
constructivist pedagogy. They seemed actively isolated, and some of the teachers interviewed 
commented on the fact. In Ambler Hill, teachers did not use the phrase “older teacher” to indicate 
resistance. And, in fact, an older teacher interviewed there said flatly though not trained in the 
“new methods,” she aimed to be part of the change and not part of the resistance. And other 
teachers indicated change (which pointed to classroom features that the evaluator could and did 
observe) in the practice of the school's supposedly “most traditional” teachers. In Western Ridges 
and Upper Ridges, older teachers were not only actively involved, but served prominently on the 
Project leadership teams. 

Finally, Ambler Hill teachers did not indicate the communications or information 
problems so frequently reported by teachers in Ambler Valley. All the teachers indicated a degree 
of vmderstanding of or involvement in the sorts of changes valued by the Project. None indicated 
that they didn't vmderstand “the big picture.” In short, professional isolation seemed less common 
in one distract than in the other. This seemed to be the case in the other districts, as well, though 
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the less extensive site- visit schedule did not permit this level of inference in those districts. 

Five possible indicators of organizational capacity characterize observed differences in the 
two case-study demonstration sites. Based on field notes and the experience of the site visits to 
those districts, these are the possible indicators: (1) stability of staffing; (2) perceived efficacy of 
leadership; (3) quality of morale; (4) professional isolation; and (5) perceived information flow. 
Serendipitously, the last four of these indicators are reflected to some degree in the Project- 
relevant measure of organizational capacity included in the cross-site teacher survey that was 
derived from a validated research instrument. As readers will see, that instrument yielded two 
factors: (1) ideological basis of collaboration and (2) actual face-to-face collaboration. 

Rural culture . This heading is, of course, a misnomer because a generalized rural culture 
can hardly be said to prevail in the U.S. any longer. Agricultural is no longer economically 
defining for rural America, and among the Project demonstration sites at least one enjoys a strong 
manufacturing economy (Ambler Valley), one is located in an officially declared “persistent 
poverty county” (Western Ridges), and one— the poorest in cross-sectional data— is so notably 
agrarian that an Amish school was established on a farm there during 1999-2000 (Ambler Hill). 

At the same time, and perhaps ironically, a generalized “American school culture” has 
frequently been said to exist with respect to schools in the U.S. Given the pervasiveness of the 



Public schools almost anywhere tend to resemble one another quite strongly and to share many 
characteristics. This culture is said (e.g., by David Tyack, 1974) to be founded on an ideal called “the one 
best system” or “the factory model.” Most schools, including those served by the Project, are founded on, 
and still to a large extent operated on, this model. The model includes such features as: age-graded 
classrooms; primary-, middle-, and high schools; standardization of practice (also known as “best 
practice”); statewide regulations for management, student attendance, teacher employment, curricula, and 
pedagogy (features of modem bureaucratic regulation, in short); and professional expertise (e.g., 
educational researchers and evaluators, curriculum development specialists, and the full range of 
certifications and endorsements that structure occupational roles in education). This generic culture of 
schooling has been examined and explained, in varying terms, by many scholars writing in the past three 
decades. Raymond Callahan's class. Education and the Cult of Efficiency (1962) is an early example; 
Stephen Arons' Short Route to Chaos (1997) is a very recent one. 
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generic “American school culture,” attempts to discover a “rural character” in schools located in 
rural places are often abortive. 

The institution of schooling seldom harbors rural purposes, as Chancellor Chu's comments 
at the December 1 999 Project conference indicated. Contemporary themes of schooling as 
articulated at the highest levels of government concern globalization, economic competition and 
restructuring, the importance of high levels of educational attainment, and applications of 
technology relevant to those themes. 

Such observations may seem remote from the concerns of the Project. Rural residents in 
Southeast Ohio, both newcomers and long-timers, however, nearly always voice strong 
appreciation for place-home, family, animals and crops, natural setting-as well as the values, 
skills and ways of encountering the world unknown and inexperienced in cities and suburbs. 
Interviews in Ambler Hill were summarized at length specifically to give voice to these concerns 
within the Project evaluation. These themes are strongly reflected in the academic programs of 
only a small minority of rural schools in the U.S. Including such themes in rural education 
requires a concerted effort of professionals and rural citizens that, for the most part, seems to run 
contrary to most state and national purposes. The contrariness of purposes has more to do, 
perhaps, with rhetorical contradictions than material contradictions, though a shortage of teacher 
time is clearly in play. Impoverished rural districts become literally preoccupied with the 
minutiae of adhering to accountability guidelines, engineering improved test scores, and 
organizing consistent efforts towards these ends in their schools and classrooms. 

A few teachers interviewed, however, expressed considerable concern about 
disconnections between school and community. For them (Cynthia Owsley, in Western Ridges, 
was among these teachers), academic performance is richly problematic because they believe very 
real academic accomplishment may not be evident in academic performance. This complex view 
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is completely consistent, incidentally, with constructivist perspectives on teaching and learning. 
Thus, the issue of place remains relevant in the minds of rural teachers even when educational 
organizations respond conceitedly to state and national rhetoric. 

Staffing is a key rural issue in Appalachian southeast Ohio. Tables 1 and 2, in fact, make 
this point rather clearly. Students are a source of rural jobs, and they are most intensely a source 
of such jobs, in general, in the poorest rural districts. Moreover, as those Tables show, the ratio of 
professional (i.e., local elite) to non-professional (i.e., local working class) jobs can actually favor 
the latter in poor rural school districts. Jobs can be distributed “rationally” according to 
professional standards and procedures, or they may be distributed more in line with attachments 
perceived by national professional norms to be irrational (e.g., family, political patronage, and 
other informal networks, see Duncan, 1999 for such a view). The evaluation did not develop data 
on this issue, and cannot make any observations about the balance between “rational” and 
“irrational” selection procedures in any of the demonstration sites. 

Staffing of rural schools is a key issue because the norms of practice imply that schools 
anywhere can (or can be expected ideally) to select candidates from a very wide pool of 
alternatives. Even in wealthy suburbs, reality falls short of the ideal. But in rural districts the 
ideal is much less attainable, and perhaps inapplicable. Rural districts employ local people, 
usually long-time residents, overwhelmingly. Outsiders employed by demonstration sites are rare, 
and 80 to 90 percent of such “outsiders” seem to reside in neighboring school districts. 



The evaluator does not necessarily believe that “rational” procedures are preferable to“irrational” 
procedures, as will become apparent shortly. Both forms can and do function to the detriment of students, 
but both can and do function to the benefit of students. For instance, an outsider (previously employed by 
a neighboring district) who, in August 2000 had just been employed by the Ambler Hill district for 2000- 
2001, told the evaluator that she took a position with Ambler Hill because the staff “really cared about 
kids. My former district,” she said, “would never have been involved in something like this [that is, 
something like Project REAL].” Nonetheless, in Ambler Hill, with this reputation of struggle on behalf of 
students and community, the web of kinship is tight among employees, and one would anticipate that this 
“irrational” element would influence employment prospects in the district. (This inference is speculative: 
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Reforming rural schools principally by importing “new blood”“Cither for teaching or 
administrative positions-is not a very workable strategy; local residents will remain the most 
logical candidates in rural schools, barring dramatic centralization of teacher assignment to 
regional, state, or federal agencies (an almost unthinkable option in the U.S.). 

Aside from employment, the other features of schooling operated on standardized state or 
national models are fraught with similar, endemic, contradictions when they play out in rural 
locales. The supposedly arm’s-length role of religion, the abstract ideal of bureaucratic 
impartiality, curriculum development, and concepts of pedagogy-all these, in rural schools, dimly 
bear the stamp of rural existence, even when rural communities and educators do not engage in 
open resistance to state- or nationally-imposed mandates. 

In some districts, the evaluator did hear educators express skepticism about whether or not 
“all [rural] children really needed” to attend college. As noted above, Abe Franklin, certainly a 
member of the “middle class,” has a son who, though passing all five Proficiency Tests in both 
the ninth and twelfth grades, is choosing not to attend college, with his father’s blessing. Perhaps 
there is something of a rural outlook in the ease with which Abe, unlike most cosmopolitan (i.e., 
non-rural) middle class acquaintances of the evaluator, countenances his son’s decision. 

In some districts that are classified as “rural,” of course, the rural character is weakening. 
Ambler Valley is a distinct example. Interviewees there told the evaluator that all the students in 
the district, within the span of their own employment, once came entirely from farm families and 
from factory families. Today, few farm families remain; they have been replaced (perhaps 



No data about such influence was gathered by the evaluator and none was offered by interviewees.) 
Perhaps in rural districts what is needed is leadership that mediates “rational” and “irrational” influences 
on employment for the benefit of both students and community. Such a process of mediation is 
hypothetically sustainable in Ambler Hill, but not (yet) in evidence in Ambler Valley. 
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displaced) by professional families, two-career families.^^ On the other hand, several of the other 
districts (certainly Ambler Hill and possibly Upper Ridges and Western Ridges) retain markedly 
agrarian features and lifeways (several have Amish enclaves, for instance), but the incomes of 
only a very few families any longer derive principally from farming. Both teachers and the 
community members interviewed in Ambler Hill described their current involvement with 
farming, animals, and crops as richly meaningful and rewarding. Several of those interviewed 
(among both faculty and community members) had moved to rural locales in part to take up (or to 
return to) a life of such meanings. 

Project REAL stands somewhere on the border between local responsiveness and state and 
national mandates. The Project occupies interesting territory and enjoys an ambiguous standing 
between funders and participants. Participants initially, and on the basis of prior experience, 
expected (like students encountering “hands-on” pedagogy for the first time) to be given exact 
instructions about how to complete their work. The Project is true to its intentions in refraining 
from the provision of these instructions. Participants are encouraged to make their own decisions 
in accord with their own understandings of reform. 

Too often, however, it seems to the evaluator. Project teams as a whole seem to be making 
their decisions on the basis of default values that arise from the generic culture of American 
schooling. Individuals on the teams, however, as noted previously, often act autonomously in a 
thoughtful, self-reflective manner. None, however, is aware of models of rural education that 
might bring rural purposes to curriculum and pedagogical action. Jamie Jackson is a partial 
exception to this rule, because her Math Lab incorporates what might be called “intrusions” into 
the real world of the community. The extension of this responsiveness into a community 



Interestingly in Ambler Valley, the family incomes of those with children are about 20% higher than 
those families without children. This situation is contrary to the national trend and perhaps indicates a 
contrast between older, poorer residents and newer, more affluent in-migrants to the district. 
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celebration--the stained glass auction—could almost be predicted from Jamie’s commitments to 
cultivating conceptual understanding simultaneously with an appreciation for the “real-world” 
presence of mathematical ideas and knowledge. The event was, of course, apparently unique 
among Project-related activities in 1999-2000. 

Teachers' knowledge of educational change and reform . This issue clearly relates directly 
to the fifth-listed theme described previously (“experiencing the ambiguities of change”). The 
issue focuses on teachers' understandings of educational change and reform. In contrast with 
teachers, many administrators are accustomed to thinking of schooling in organizational, and 
sometimes even institutional, terms. Teachers are not accustomed to think this way, and they 
generally view their organization and the institution of schooling itself through the lens of 
classroom practice— most particularly their own classroom practice. The view is dangerously 
myopic, and the myopia is understandably difficult to shake. 

Changes that come to this realm (teachers’ own forceful experience of the classroom) 
from outside it may arrive as mandates, or they may be insinuated by colleagues or by their own 
self-guided professional development. Most commonly, the changes may arrive in some 
combination of influences. 

But teachers rarely think of, nor do they have much knowledge of, the political, historical, 
economic, or ideological conditions of educational reform. Their undergraduate classes dealing 
with these topic probably seemed abstract, remote, and hopelessly impractical at the time. Similar 
treatments at the master’s level are more rare, but the forceful experience of the classroom 
probably renders them even more irrelevant-seeming to teachers. Thus, many teachers remain 
skeptical of the wider issues of school and organizational reform, with the basis of their 
skepticism an organizationally limited base of knowledge and experience. 

In this light, it is not very surprising that a some interviewees confessed that they did not 



understand “the big picture” of reform.^'* They were, in these cases, most likely referencing the 
“big picture” of reform in their own districts, but, naturally, the big picture is larger still. Some 
teachers, one must conclude, view reform in the role of recipient rather than of creator. 

The role of recipient of reform is, unfortunately, reinforced institutionally. Public school 
teachers nearly everywhere work xmder conditions in which changes are often not just projected, 
but promised by such authorities as governors, legislatures, chief state school officers, 
superintendents, and principals. The most extreme promises for change tend to come from the 
highest levels. This dubious rhetoric of change combined, in particular, with the tendency of 
high-level authorities to impose or mandate change, cultivate in teachers several unfortunate 
expectations: (1) change is something done to them by others; (2) someone knows what the 
future is supposed to be; (3) the will of teachers coxmts for little except as resistance to change. 

These comments of the evaluator's, especially the last one, which sums up this feeling of 
some participants, seem to center on teachers' comparative lack of familiarity, not so much (as 
they might themselves insist it did!) with the particularities of the Project's big picture, or even 
with the overall intentions for educational reform held by leaders in their particular district, but 
instead with the sorts of contests over educational reform that might manifest themselves in local 
schools, districts, and classrooms. Who is doing what to whom? Why? How do I fit in? These 
are questions that concern the vast majority of teachers who are not involved in Project decision 
making. Even those in Project leadership roles have these questions, but they are better 
positioned, and perhaps better disposed, to provide their own answers to such questions. 

Lacking the big picture of which interest groups desire what sorts of reform and for what 



On the other hand, teachers like Cynthia Owsley, Jamie Jackson, Denzil Kline, Rita Casto, Teresa 
Cotton, and Ben Lively all showed some appreciation for “the big picture,” even when sometimes denying 
such appreciation. Some of these teachers showed an uncanny sensitivity to critical, problematic issues, 
as reported in the case studies and interview summaries. 
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reasons, and without much opportunity to decide for themselves, or among themselves, what 
makes sense— most individual teachers are perhaps caught, especially when in charge of designing 
their own reform plans, with obscure misgivings, regrets, and resentments. Again, the evaluation 
reports many exceptions to this speculation. 

Anyone intending change, or harboring good intentions for organizational or institutional 
futures, needs to appreciate the professional ambiguities of teachers’ worklives. Teachers 
themselves are not, actually, in a good position to harbor much appreciation of this sort! 

First, they are generally the recipients of administrative directives of one sort or another. 
Such directives (e.g., policies, regulations, and memos directed at them, and paperwork required 
of them) are often seen as distracting, burdensome, and inescapable. Compliance is not optional; 
it is a mostly an unwelcome chore. This feature of teachers' worklives simultaneously 
circumscribes their autonomy and their experience of ambiguity in their realm of practice (i.e., the 
classroom). 

Second, teachers work to establish classroom routines that feel, to them individually, most 
productive for them and their students. For teachers, this level of comfort describes “what works.” 
Internally, then, teachers strive actively to reduce the ambiguities of their own classroom practice. 
Order is very important in schools, and teachers are the frontline in the maintenance of such order. 

The irony for Project REAL is that the sort of change that teachers generally experience is 
the mandated change. Proficiency testing, accompanied with detailed procedures and regulations 
and classifications and sanctions, is just such a change. Teachers and administrators are told what 
to do, how to do it, when to do it, and (not very elaborately) why to do it. Most are skeptical and 
resentful. Some are fearful. Even those who believe that, overall, accountability is beneficial, 
retain substantial reservations that concern the possible evil done by unintended consequences. 

The Project enters this playing field with a very different message, but one that is more 



consistent with many teachers’ professional preparation, with their intentions in choosing 
education as a work, and with their aspirations for professional growth. The burden of reform, 
both mandated and self-motivated, actually falls on them. The Project seems to the evaluator to 
provide some of the support-coupled with an autonomy of local action and respect for the 
dilemmas of teaching— that they so much need if they are to negotiate the complexities of 
organizational change required to support more authentic teaching and learning. 

Cross-Site Teacher Survey 

The January 2000 mid-project planning document (see Appendix B) called for a cross-site 
teacher survey. Although the evaluator visited all demonstration sites more than once, he focused 
his on-site work after January 2000 most closely the two case study districts, the results of which 
work is summarized in the preceding section of this report. The cross-site survey was adopted as 
a strategy to give a broader view of the project, whereas the case studies were intended to provide 
depth. Surveys were completed in mid-May 2000, by all teachers in the grade levels served by 
the Project (that is, whether or not they taught mathematics or science) in each of the five 
districts. Overwhelmingly, this included teachers with assignments grades 5-8. Only eight of the 
136 respondents did not have assignments at this level. 

Purpose and Design 

The intended purpose of the survey, as outlined in January 2000, was to assess, across all 
five demonstration sites, teachers' readiness to initiate or accept change, specifically the sorts of 
changes desired by the project. At that time the evaluator and the project director speculated that 
differences among the districts might distinguish their capacity to benefit from Project activities 
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and involvement. They cross-site survey was conceived to provide a systematic assessment of 
this surmise than was possible within the design of a purely responsive evaluation. During 
development of the survey instrument and contingent on further discussions between the evaluator 
and the project director, the purpose was amended to include provision for a summative measure 
of first-year Project impact (considered at some length, below under “Findings.”) 

Teachers were specified as respondents since an important aim of the project was to 
influence classroom instruction. In addition, teachers were major participants and some teachers 
were Project leaders in their districts. 

The evaluator designed the survey instrument in consultation with the project director, 
who reviewed interim drafts and approved the final draft. In its final form, the survey (see 
Appendix C) contained three sections: background (6 questions), project (6 questions), and the 
district (10 questions) . 

The 6 background questions pertained to district identity, program-relevant teacher 
characteristics (years teaching, subjects taught, grade levels taught, opinion toward “hands on 
learning,” and change in view of hands on learning during the past year). The final question was 
subsequently used in the construction of the previously mentioned summative measure of first- 
year project impact. Question 5 asked about the influence of Project REAL or “other recent” 
mathematics and science professional development activities 

The 10 questions about the district were adapted from a longer validated instruments^ 
developed by AEL, Inc., an educational research and development firm that holds a variety of 
federal and state contracts. These questions were included to address the issue of organizational 
capacity to engage in the sorts of change valued by the Project (see Appendix C). The Project, 

The evaluator selected and modified items from an instrument whose use was reported in Cowley, 
Nielsen, and Ceperley (2000). The evaluator thanks the authors for permission to adapt items from their 
instrument. 



committed to constructivist views of teaching and learning, not only supports teachers in helping 
students construct knowledge in math and science (in accord with state and national 
expectations), but also supports teachers in constructing their own professional practice along 
similar lines. Such construction is based on professional collaboration, precisely the construct 
measured by these 10 items. Part of the aim in including these 10 items was to provide a 
validated short instrument that the Project might use with districts or schools that subsequently 
became part of the project. 

Respondents' Backgrounds 

Across all sites, 136 teachers completed surveys. The number of respondents available for 
various analyses, however, is usually somewhat reduced (up to 10 fewer cases) because of 
missing data. Approximately half the teachers in the population are employed in the Western 
Ridges district, simply because that district is about 5 times as large as the smaller districts in the 
Project (see Table 2) and serves all math and science classrooms in grades 5-8. 



Table 3 



Cross-site Teacher Survey Respondents by District 



District 


Frequency 


Percent 


Western Ridges 


69 


50.7 


Upper Ridges 


10 


7.4 


Ambler Hill 


19 


14.0 


Lower Ridges 


13 


9.6 


Ambler Valley 


25 


18.4 


Total 


136 


100.0 



Of the 136 responding teachers surveyed, 8 reported that they taught grade 4 (6 in Ambler 
Hill, and one each in Upper Ridges and Western Ridges); 15 teachers reportedly taught grade 9 or 
above (10 in Western Ridges,^* 4 in Lower Ridges, and 1 in Upper Ridges). Only 8 of the 136 
teachers, however, had teaching assignments that did not include some instruction in grades 5-8 (3 
in Ambler Hill, 2 each in Western Ridges and Lower Ridges, 1 in Upper Ridges, and none in 
Ambler Valley). All teachers (n=3) who taught only grade 4 were employed by Ambler Hill; the 
remaining 5 teachers taught only grades nine or above. 

Analysis of variance comparisons suggest that Ambler Hill respondents involved 
significantly more teachers with some or all assignments in grade 4, while Lower Ridges 
respondents involved significantly more teachers with some or all assignments in grade 9 or 
higher. This finding probably reflects the nature of site leadership, with the Ambler Hill 
leadership team rooted in professional experience at middle- and elementary-school levels, 
whereas the Lower Ridges leadership team has its base of professional experience at the high 
school level. 

In all districts, the modal response for years spent teaching was category “4” (11-20 years). 
Analysis of variance showed no significant difference between districts on this variable. Across 
the entire group, 62.5% of respondents had taught for at least 1 1 years. Eight percent had been 
teaching for 30 years or longer, whereas 13.5% had taught for fewer than 3 years. 

Of the 136 teachers, 86 taught math or science (or both), whereas 50 taught neither math 
nor science. The proportion of teachers with assignments in math or science (or both) varied 
significantly by district. The Ambler Valley site returned the greatest number of teachers (all with 
assignments in grades 5-8) who did not teach either mathematics or science (68%). This 
circumstance may well be the result of a higher degree of teacher subject-specialization in Ambler 



“ Western Ridges maintains a number of 7-12 high schools, so that assignments crossing middle- and 
high-school levels would be conveniently accomplished. 



Valley. Across all five sites, however, about two-thirds of teachers reported that they taught math 
or science, or both. 

The background section of the survey also asked respondents two questions about their 
opinion of “hands on learning””— their opinion at the time of completing the survey and the 
change (comparative degrees of positive or negative) in their view during the course of the past 
year. Table 4 reports frequencies for these two items. 



Table 4 — Views of “Hands-on Learning” 



Opinion of “hands-on learning” now (May 2000)? 



response 


Number of teachers 


percent 


somewhat unfavorable 


2 


1.5 


lukewarm 


15 


11.0 


favorable 


60 


44.1 


very favorable 


59 


43.4 


Total 


136 


100.0 



Change in view of “hands-on learning” in past year (since May 1999)? 



response 


Number of Teachers 


percent 


much less positive 


1 


.7 


somewhat less positive 


2 


1.5 


has not changed 


75 


56.0 


somewhat more positive 


28 


20.9 


much more positive 


28 


20.9 


Total 


134 


100.0 


missing 


(2) 






136 





This phrase was used as a more accessible alternative to “constructivism” or “constructivist methods of 
teaching and learning.” 
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Subsidiary analyses (not reported in tabular form), showed a very interesting result. A 
slightly higher (but statistically nonsignificant) proportion of teachers without any teaching 
assignments in math or science reported positive change in their opinion of hands-on-leaming 
over the course of the 1999-2000 school year. Among such teachers, 42 percent reported such 
change, as opposed to 40 percent of m*ath-science teachers. In fact, in ancillary analyses not 
presented in the Tables, both groups exhibit similarly positive views of hands-on learning (about 
87 and 88 percent, respectively, reporting “favorable” or “vary favorable” views). 

Respondents' Answers to Project Questions 

Results (frequencies) for the six Project questions appear in Table 5. Findings are 
discussed following presentation of the Table. 



Table 5: Frequencies for Project Questions 



Project Real Team Member? 





Number of teachers 


Percent 


no 


108 


80.6 


yes 


19 


14.2 


unsure 


7 


5.1 


missing 


(2) 




Total 


136 


100.0 
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(Table 5, continued) 



Degree of Involvement in Project REAL? 





Number of teachers 


Percent 


not at all 


64 


47.4 


a little 


28 


20.7 


some 


20 


14.8 


a lot 


16 


11.9 


don't know 


7 


5.2 


missing 


(1) 




Total 


136 


100.0 



Ever heard of Project REAL? 





Number of teachers 


Percent 


no 


8 


5.9 


yes 


128 


94.1 


Total 


136 


100.0 



Frequency of use of national or state materials for classroom planning? 





Number of teachers 


Percent 


not at all or not 
familiar with 


34 


25.6 


less than monthly, but 
sometimes 


28 


21.1 


at least monthly 


22 


16.5 


at least weekly 


26 


19.5 


almost daily 


23 


17.3 


missing 


(3) 




Total 


136 
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(Table 5, continued) 



REAL (or other M/S professional development) influence on thinking now? 

(May 2000) 





Number of teachers 


Percent 


not at all 


40 


29.9 


a little 


33 


24.6 


somewhat, but 
more than a little 


21 


15.7 


quite a lot 


30 


22.4 


enormously 


10 


7.5 


Total 


134 


100.0 


missing 


(2) 




Total 


136 





REAL (or other M/S professional development) influence on teaching now? 

(May 2000) 





Number of teachers 


Percent 


not at all 


47 


34.6 


a little 


21 


15.4 


somewhat, but 
more than a little 


29 


21.3 


quite a lot 


27 


19.9 


enormously 


12 


8.8 


Total 


136 


100.0 



As anticipated, the overwhelming majority of respondents (80.5 percent) reported that they 
were not members of Project REAL teams. A few respondents (n=7, or 5.1 percent) were not 
sure if they were team members. Since all team members in the demonstration sites were, in fact, 
quite sure of their designation as team members, the proportion of respondents who were not 
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team members is most properly reckoned as 86 percent (compare to the first panel in Table 5). 

About half the respondents (compare to second panel in Table 5) reported that they had at 
least “a little” involvement with Project REAL, and about 60% of those who reported any amount 
of involvement reported “some” or “a lot” of involvement. 

In interpreting these results, one must recall that 50 of the 136 respondents taught neither 
math nor science. Among respondents with assignments in mathematics or science, however, 
only 37% reported no involvement with the Project. Independent t-tests of means between math- 
science teachers and the other teachers (equal variances not assumed) showed the mean difference 
of +.74 to be highly statistically significant (p<.001) in favor of respondents with math-science 
assignments. The effect size for this difference (that is, the difference in standard deviation units 
of the group of respondents without mathematics and science assignments) was about 1.0-a very 
strong effect size Approximately 30% (n=14) of teachers without any assignments at all teaching 
mathematics of science were involved in Project activities. In fact, about 1/3 of these teachers 
(n=4) reported substantial involvement (“some” or “a lot”). Thus, even teachers not specifically 
targeted by the Project reported some degree of involvement.. The evaluator concludes, therefore, 
that math and science teachers were substantially more involved in Project REAL activities 
than other teachers, but that many other teachers were also involved in Project activities. 

Quite predictably, perhaps, the involvement of Project team members exceeded that of 
other groups by a wide margin. Independent t-tests (equal variance not assumed), put the level of 
involvement of Project REAL team members in Project activities at two standard deviations 
above those respondents with assignments in mathematics or science who were not team members 
(that is, team membership had an effect size of greater than +2.0 on involvement in Project 
activities). This highly significant statistic (p<,001) suggests an overall high level of 
engagement and commitment to the Project of team members who were teachers. 
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Of the 136 respondents, only 8 reported never having heard of Project REAL. Half of this 
group had assignments in mathematics or science, but half did not. Half (a proportionate number) 
were from the large demonstration district. Western Ridges. All but one in this group of 8 
respondents had teaching assignments in seventh or eighth grade. The other respondent had a 
high-school teaching assignment. 

Approximately 75 percent of all respondents report that they use national or state materials 
to prepare classroom activities or lessons at least sometimes, whereas nearly 40 percent use them 
at least weekly. These statistics, too, are significantly higher for math-science teachers than for 
others, and very much higher for Project REAL team members; only 15 percent of math-science 
teachers report no familiarity with such materials, and more than 50 percent report using such 
materials weekly (and half of these report daily use). Interestingly, responses from Project REAL 
team members do not indicate that team members use such materials more frequently than other 
math-science teachers. 

Across all respondents, about 30 percent report that recent professional development 
efforts in mathematics and science have influenced their thinking and their teaching “quite a lot” 
or “enormously.” Again, these proportions are significantly higher among math-science teachers 
and Project REAL team members, with an effect size of about +1.0 favoring both the math- 
science teachers and the Project REAL team members (as compared to all other respondents in 
each case). Again, Project REAL team members do not report substantially stronger influences 
on their thinking and teaching than either (1) all respondents or (2) math-science teachers. 
Respondents without any teaching assignments in math or science, however, also reported being 
influenced in their thinking and actual teaching by “Project REAL or other recent professional 
development activities related to math and science.” Half of these respondents reported influence 
on their thinking and about 40 percent reported influence on their actual teaching. Again, this is 



evidence that the Project is having schoolwide influence beyond its focal domain in 
mathematics and science. 

Respondents' Descriptions of Professional Community in Their Districts 

These 1 0 items were administered as a measure of respondents' perceptions of schoolwide 
professional community in their districts for possible use as a measure of capacity for change 
consistent with the sorts of change encouraged by the project. 

The instrument from which these 1 0 items are adapted is The Teacher, School, and 
School Organization Questionnaire, described in Cowley et al. (2000). That lengthy instrument 
included three parts. Items adapted for this evaluation belonged to the third part, Schoolwide 
Professional Community (SWPC). The SWPC contained 30 items, representing five subscales. 
The evaluator selected and modified two items from each subscale. An alpha reliability of .88, 
for n=132 cases with valid data, was obtained, exactly same as the average of the reliabilities of 
the five subscales of the full-length instrument (see Cowley et al., 2000, p. 5). The short version 
of the SWPC, then, has the technical characteristics adequate to measure the underlying construct 
(i.e., professional community). Factor analysis of the 132 valid cases produced two factors, each 
loading most heavily on five items, as shown in Table 6. 
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Table 6 - Factor Structure of the 10=Item SWPC Adaptation 



items 



factor loadings 



factor 1 - 



agree on expectations 
commitment to authentic instruction 
focus on how well students learn 
degree of cooperative effort 
share core beliefs 



.87 
.81 
.79 
. 76 
.74 



factor 2 - 



often visited other classrooms 
colleague regularly observes me 
administrator gives comments 
discuss with one another 
share useful suggestions 



.80 
.80 
.72 
. 69 
.59 



(also loaded .54 on factor 1) 



These factors seemed to capture two dimensions of schoolwide professional community. 
Factor 1 most evidently concerns agreement on common beliefs, expectations, and commitments- 
-in other words, the ideological dimension of professional community. Factor 2, by comparison 
most evidently concerns interaction among professionals in such a community, visiting one 
another's classrooms and discussing instructional and curricular matters with one another— in other 
words, the dimension of face-to-face interaction. Factor scores were computed for use as possible 
variables in subsequent analyses.^* 

Analyses of variance computed by district revealed one statistically significant difference 
(p=.044) by district for the total scale score and the two factor scores, among districts on factor 1 

Factor scores have a mean of 0 and a standard deviation of 1. Thus a factor score of +2.0 is a very high 
score, whereas a factor score of -2.0 indicates a very low score. Alternatively, but with less sensitivity to 
the underlying factor, one might simply sum the items loading most heavily on each factor. The related 5- 
item scales have sufficiently strong reliabilities, despite being composed of so few items (alpha=.88 for 
the five items loading most strongly on factor 1 items and .82 for the five items loading most strongly on 
factor 2). The items could be used in this fashion in future analyses of Project REAL evaluation data. 
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(ideological dimension). Post-hoc Tukey tests revealed that the difference in factor 1 scores 
between Western Ridges (factor 1 score = -.18) and Lower Ridges (factor 1 score = +.70) was 
statistically different at p< .05. Full-scale total scores varied by district from 3 1 .6 (Western 
Ridges) to 37.3 (Lower Ridges); factor 1 scores varied from -.18 (Western Ridges) to +.70 (Lower 
Ridges); and factor 2 scores varied from -.33 (Upper Ridges) to +.24 (Lower Ridges). 

Other groupings showed significant differences on factor 2 (face-to-face interaction factor) 
in independent t-tests. The first grouping involved dividing respondents into two groups, those 
who rated their change in view of hands-on learning as “somewhat more positive” or “much more 
positive” versus others. A statistically significant effect size of +.40 on factor 2 favored the 
group (n=55) that reported substantially more positive views of hands-on learning. The second 
grouping also divided respondents into two groups, those who reported at least a little 
involvement in Project REAL activities versus those who reported no involvement. A statistically 
significant effect size of +.44 on factor 2 favored the involved group (n=64) as compared to the 
uninvolved group. 

The evaluator concludes that this short instrument has prospective practical value for the 
Project as a baseline measure and as a possible contextual or predictor variable in subsequent 
analyses. 

Findings About Project Impact 

Evidence presented thus far is suggestive of possible Project impact on a variety of school 
and classroom practices. In order to test this hypothesis further, some technically adequate 
measure of Project impact is needed. 

Fortunately, the survey instrument contains four questions that do assess such impact. 
These questions (see Appendix C for the complete phrasing) follow: 
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1. view of hands-on learning (as of May 2000) 

2. use of national and state materials to create activities and lessons (as of May 2000) 

3. change in thinking from REAL(or other recent math-science efforts) as of May 2000 

4. change in actual teaching from REAL (or other math-science efforts) as of May 2000 

To gauge the technical adequacy of this measure, the evaluator conducted both a factor 
analysis and a reliability analysis. Factor analysis extracted a single factor with an eigenvalue 
above 1 (the standard for identifying useable factors), explaining two-thirds of the variance 
among the four items, each of which load between .69 and .92 on the single factor. This singular 
factor would logically be called “project impact.” At alpha = .82, these four items show good 
internal consistency as a scale, indicating that the items together measure a single construct quite 
well (i.e., a construct that would be logically called “project impact”). The evaluator concludes 
that this measure is technically adequate to assess Project REAL impact.^® 

As the measure of project impact, these analyses employ the sum of the values reported by 
respondents on the four items described above. The measure has ample range and variation to 
serve this purpose, with a mean and median of about 20, a minimum of 5, a maximum value of 
20, and a standard deviation of about 4.“ The distribution is nearly normal (skewness = 0.326). 



This measure of impact, of course, is based on self-report information and not on data gathered by 
independent observers. The various case studies and interviews conducted by the evaluator, however, 
tend to corroborate the existence of impact as defined by this measure. 

“ It is interesting to note that just 14 percent of respondents reported that they never used state or federal 
materials to plan instruction and had not been influenced in their thinking and their teaching by REAL or 
other recent professional development in mathematics or science. That is, only a small portion of these 19 
respondents reported information that would indicate they had experienced minimal Project impact or 
none. Nearly 80 percent of these teachers reported that they did not have teaching assignments in 
mathematics or science, and nearly 75 percent reported that they had teaching assignments that included 
7th or 8th grade. In fact, about 65 percent of these teachers had upper-level assignments and did not teach 
mathematics or science. 
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The measure is used to answer six questions about Project REAL impact, as follows: 

1. To what extent does Project impact vary among districts? 

2. To what extent does Project impact vary among subdistricts in Western Ridges? 

3. To what extent does Project impact vary among other subgroups of respondents (i.e., 
those with low involvement, math-science teachers versus others, those low on factor 2 
versus others, those low of factor 1 versus others, those low on schoolwide professional 
community versus others, those teaching grades 7 or 8 versus others, those teaching 
more and less than the median number of years)? 

4. What influences predict Project impact across all respondents? 

5. What influences predict Project impact among math-science teachers? 

6. What influences predict Project impact among respondents without teaching 
assignments in math or science? 

The first three questions are answered by comparing means among the groups specified; 
the final three questions are answered using regression analysis. Results of analyses of variance 
and independent t-tests of the means tests will be summarized rather than reported in detail in 
tables. 

1. To what extent does project impact vary among districts? 

Mean scores on the summative measure of Project impact (maximum possible score is 
20, minimum is 5; the overall mean was 11.5; and the median score was 11) varied by 
district from 9.5 (Ambler Valley) to 12.3 (Western Ridges and Upper Ridges). Lower 
Ridges and Ambler Hill had mean scores of 12.0 and 10.4, respectively. Overall 
differences among districts vary significantly (p=.036). It is reasonable to conclude 
that Project impact varied significantly among districts. Impact was least in Ambler 
Hill and greatest in Western and Upper Ridges, based on this measure. 

2. To what extent does project impact vary among subdistricts in Western Ridges? 

Though the information was not sought. Western Ridges provided its completed 
surveys organized by attendance areas within the this consolidated county district. 
Central Area had a mean score of 9.4, Western Area a score of 12.2; Eastern Area a 
score of 12.2; and Southern Area a (remarkably high) score of 15.0. Overall differences 
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among Western Ridges attendance areas vary significantly (p = .024). It is reasonable 
to conclude that Project impact varied significantly among Western Ridges 
attendance areas. It is perhaps relevant to observe that Southern Area, during the 
1999-2000 school years, enjoyed the full-time assistance of a teacher-facilitator because 
of its participation in the Appalachia Rural Systemic Initiative (ARSI). 

3. To what extent does project impact vary among other subgroups? 

Those with no Project involvement versus others’. Respondents with no involvement 
had a mean group Project impact score of 9.1, as compared to 13.9 for those with at 
least “a little” involvement (equivalent to a substantial effect size of +1.55) . The 
difference was highly significant (p<.0005). 

Math and science teachers versus others’. Respondents without math-science teaching 
assignments had a group mean impact score of 9.0, versus 12.9 for those with such an 
assignment (equivalent to a substantial effect size of + 1.34). The difference was highly 
significant (p<.0005). 

Respondents low on factor 1 versus others’. The mean impact score was actually lower 
for respondents high on factor 1 (ideological dimension) than others (divided at the 
median factor score of +.21 8), 11.9 versus 11.0. The difference was not statistically 
significant (p=.277). 

Respondents low on factor 2 versus others’. The mean impact score of those 
respondents scoring above the median (+.139) on factor 2 (face-to-face dimension) was 
12.3, as compared to 10.6 for others (equivalent to an effect size of about +.45), a 
statistically significant difference (p = .016). 

Respondents low on schoolwide professional community versus others’. The mean 
impact score of respondents scoring above the median (33) was 1 1.6, versus 1 1.3 for 
those scoring below the median. The difference was not statistically significant (p = 
.679). 

Respondents teaching grades 7 or 8 versus others: The mean of respondents with 
teaching assignments in grades 7 or 8 was 10.8, as compared to 12.3 for those without 
such assignments (equivalent to an effect size of about .40). The difference was 
statistically significant (p = .039). 

Respondents teaching more and less than the median number of years: The mean of 
respondents teaching the median number of years (1 1-20) or longer was 1 1.3, as 
compared to 11.7 for those teaching fewer years. This small difference (equivalent to 
an effect size of +.12) was not significantly different (p = .526). 

The discussion now turns to answering questions 4-6. The method used to address these 
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questions is “regression analysis.” Regression analysis estimates the value of one variable (the 
dependent variable, in this case “project impact”) from a set of predictor variables (also known as 
“independent variables”). 

This method is especially useful because it permits the evaluator (1) to account for the 
proportion of variation in the dependent variable associated with the predictor variables arid (2) to 
specify which variables are statistically significant unique influences on Project impact. 

Three regression equations were constructed, each using the same set of independent 
variables,®' to predict Project REAL impact (using the 4-item scale of impact). The three analyses 
thus differ only by the subgroup selected for analysis: (1) all respondents (n = 136); (2) 
respondents with assignments to teach mathematics or science; and (3) respondents without 
teaching assignments in mathematics or science. The latter two analyses were performed to 
distinguish possible influences among the target group of the Project. Results of the three 
analyses appear in Table 7. Sample sizes for these analyses are somewhat less than for other 
analyses since analyses were possible only for those respondents whose records included valid 
data for all variables. 



The seven independent (or “predictor:) variables were as follows: (1) years teaching, (2) opinion of 
hands-on-leaming as of May 2000, (3) Project REAL team membership, (4) level of involvement in 
Project activities, (5) being a math-science teacher, (6) having a teaching assignment at grade 7 or 8, and 
(7) factor 2 (face-to-face dimension of schoolwide professional community). For the equations among 
math-science teachers only and other teachers only, variable #5 was, of course, omitted. 

117 




IcS 



Table 7 - Overall Prediction Equation 



Variable 


B 


B 


t 


P 


(Constant) 


.459 




.353 


.725 


Change in view of hands-on 


2.221 


.475 


7.528 


.000 


Local REAL team member 


.865 


.078 


1.010 


.315 


REAL Involvement 


.968 


.259 


3.167 


.002 


Face-to-face (factor 2) 


.322 


.078 


1.354 


.179 


Shared beliefs (factor 1) 


-.353 


-.081 


-1.508 


.134 


Math-science teacher 


2.695 


.315 


5.187 


.000 


7-8 teacher 


-.086 


-.010 


-.183 


.855 


YRS TCHG 


-.181 


-.065 


-1.166 


.246 



N = 1 13 (number of cases with valid data for all variables) 

= .684 (equation explains 68.4% of observed variance in Project REAL impact) 

Prediction Equation for Math-Science Teachers 



Variable 


B 


B 


t 


D 


(Constant) 


2.466 




1.403 


.165 


Change in view of hands-on 


2.582 


.592 


6.606 


.000 


Local REAL team member 


.284 


.031 


.272 


.786 


REAL Involvement 


.962 


.279 


2.540 


.014 


Face-to-face (factor 2) 


.168 


.043 


.541 


.590 


Shared beliefs (factor 1 ) 


-.425 


-.099 


-1.302 


.198 


7-8 teacher 


.142 


.018 


.224 


.824 


YRS TCHG 


-.347 


-.132 


-1.717 


.091 



N = 71 (number of cases with valid data for all variables) 

R^ = .61 1 (equation explains 61.1% of observed variance in Project REAL impact) 

Prediction Equation for Other Teachers 



Variable 


B 


Beta 


t 


P 


(Constant) 


2.367 




1.360 


.183 


Change in view of hands-on 


1.214 


.333 


2.496 


.018 


Local REAL team member 


3.071 


.156 


1.318 


.196 


REAL Involvement 


1.460 


.384 


2.349 


.025 


Face-to-face (factor 2) 


.451 


.144 


1.198 


.239 


Shared beliefs (factor 1 ) 


-.308 


-.100 


-.985 


.332 


7-8 teacher 


-.816 


-.128 


-1.148 


.259 


YRS TCHG 


.168 


.076 


.674 


.505 



N = 42 (number of cases with valid data for all variables) 

R^ = .587 (equation explains 58.7% of observed variance in Project REAL impact) 
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Three findings emerge from the results reported in Table 7. First, and most remarkably, 
all three prediction equations account for a substantial amount of variance in Project REAL 
impact: from about 60% (for “other” teachers) to about 70% for all teachers together. For 
any sort of regression analysis in the social sciences, this is a strong result. For instance, studies 
of influences on student achievement rarely account for this much variance in achievement. 

Second, in each analysis the prediction equations show a consistent pattern of influence 
among predictor variables. Level of involvement in Project activities and change in view 
(over the 1999-2000 school year) of “hands-on” learning predict most of the variation in 
project impact.®^ Regression analysis gives the unique influence of predictor variables, so that it 
is correct to conclude that these variables exert a separate influence from one another. 

Third, the overall equation, as compared to those for the contrasting groups of teachers, 
shows that having a teaching assignment in mathematics or science has a predictably positive 
influence on project impact— predictable since these teachers constituted the focal group for the 
Project. It would, after all, seem appropriate and desirable for the focal group of any project to 
exhibit the greatest impact of project activities. 

Nonetheless, the third panel in Table 7 (that is, teachers without any teaching assignment 
in mathematics or science) also demonstrates again that the impact of Project REAL extended 
well beyond its focal group. Teachers who somehow became involved in Project-sponsored 
activities regardless of their teaching assignment exhibited project impact. The stronger their 
involvement, the stronger the impact; the more positive their change in view of hands-on learning, 
the stronger the impact. 



“ Recall that “project impact” is defined as the sum of final view (May 2000) of hands-on learning, use of 
state and national materials to create activities and plan lessons, change in thinking and teaching due to 
professional development activities. 
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One nearly significant result might be mentioned, in view of site-visit data that indicated 
that some respondents regarded older teachers as less likely to benefit from Project involvement. 
In terms of impact data, none of the analyses revealed years teaching to predict project impact. 
Among math-science teachers this variable shows up as a possibly weak influence, but it does not 
a level of statistical significance. This means that the observed weak influence might just as well 
be due to chance. Older teachers, according to the data analyzed here, cannot be understood 
as being less likely to benefit from Project activities than other teachers. 

Summary of Findings and Conclusions from the Cross-Site Teacher Survey 

The findings from the preceding analysis are recapitulated below, with related conclusions 
presented in boldface. For details, please refer to the preceding tables and discussion. 

About one in seven respondents (14%) was a member of a local Project REAL team, and 
about one in two respondents (50%) reported some degree of involvement in Project activities. 
Among those with involvement nearly 3 out of 5 respondents (60%) reported “some” or “a lot” of 
involvement. Virtually all respondents (94%) had heard of Project REAL, whether or not they 
reported being involved in Project activities. These demographic data indicate that the Project 
is strongly in evidence in the Project districts. 

Comparison of the involvement of local Project Team members with the involvement of 
others revealed much higher levels of involvement for team members. This result is consistent 
with the conclusion that Project REAL leadership teams are highly engaged across all 
Project sites. This is strong evidence of the commitment of local teams. 

Between 3 in 4 (74.4%) and 2 in 3 (65.4%) of respondents indicated positively that they 
used state or national standards to plan instruction, or that Project REAL or other recent 
professional development activities had influenced their thinking about instruction, or had 
influenced their actual teaching. These data indicate clearly that the Project is helping to 
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encourage among all teachers the sorts of dispositions valued in current reform efforts. 

The survey included teachers with no assignments in mathematics or science as well as 
math-science teachers. In fact, nearly 2 of 5 respondents (36.7%) taught neither math nor science. 
The difference in level of involvement between math-science teachers and others was equivalent 
to an effect size of about +1 .0. Approximately 30% of teachers without assignments in 
mathematics or science were involved, some at a very high level. Although math and science 
teachers were substantially more involved in Project REAL activities than other teachers, 
those other teachers were also involved in Project activities to a measurable extent, some at 
the highest levels. 

Interestingly, respondents without any teaching assignments in math or science reported 
being influenced in their thinking and actual teaching by Project REAL or other recent 
professional development activities related to math and science. Half of such respondents 
reported influence on their thinking and about 40 percent reported influence on their actual 
teaching. These findings strongly suggest that the Project is having schoolwide influence 
beyond its focal domain in mathematics and science. 

The evaluation defined, and posed six questions about, project impact. The analyses 
(comparison of means and regression equations) yielded a number of statistically significant 
findings. In tests of means, project impact was shown to vary most significantly by (1) subject 
assignment®^ and (2) involvement®'* in Project REAL activities. ®® A remarkable proportion of the 
variance in project impact was accounted for in subsequent regression analysis by three variables; 
(1) level of involvement in Project REAL, (2) change in view of hand-on-leaming, and (3) having 
a math-science teaching assignment. Other likely influences (which district employed the 



Very significantly favoring math-science teachers. 

®^ (very significantly favoring those involved vs. those not involved) 

Less strongly significant differences appeared as well (see the preceding discussion). 
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respondent, years respondent had taught, respondent's teaching level, various aspects of 
schoolwide professional community, and whether or not the respondent was a local REAL Project 
team member) did not exert unique influences on project impact. Clear evidence indicates that 
involvement in Project REAL activities and ideology (i.e., developing a more positive view of 
hands-on learning) had a strong effect on the extent of project impact among individual 
teachers. 



Formative Questions for the Project 

The evaluator poses a number of formative questions for Project staff in this section. 
Questions appear under the rubric of the three critical issues, plus a miscellaneous category of less 
significant, but possibly more practical questions. The questions, needless to say, represent the 
evaluator's sense of the importance of the themes and issues, based on evaluation data and 
experience of school reform in rural areas. Some of the questions, however, can be traced very 
directly to concerns expressed by Project participants. 

Organizational Capacity 

• Should the Project assess organizational capacity in its sites? 

• If so, how might this be done with acceptable levels of validity, reliability, and 
efficiency! 

• Should the Project take steps to improve the adoption of the sorts of pedagogy it values 
in districts with jeopardized organizational capacity? 

• If so, what steps are consistent with the Project's intention to permit local actors to 
design and implement locally appropriate changes? 

Rural Culture 

• How can the Project adopt a pro-rural stance while simultaneously supporting state and 
national reform efforts that cultivate “hands-on” and “engaged” teaching and learning? 

• How can the Project provide in-service opportunities that explicitly build on rural 
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themes and issues! 



• How can rural communities become more involved in academic content in non- 
threatening and inclusive ways! 

Understanding Educational Change 

• How can the Project help a greater proportion of teachers become more active creators 
of change! 

• How can Project teachers and administrators productively share their life experiences 
of reform and change across project sites? 

• Which understandings of the history, politics, economics, and ideology of school 
reform might be most relevant to rural classroom teachers and how can such 
understandings be effectively cultivated among them? 

• How can the Project help rural educators develop a productively skeptical view of 
educational change and reform? 

Questions Reflecting Themes and Other Issues 

• In small districts (enrollment less than about 1,500) should all middle-school math and 
science teachers be members of the local Project team? 

• How can the Project help teachers at Project sites network more effectively with one 
another (e.g., by arranging visitations or classroom observations among districts, 
holding specialized workshops for subsets of participants, providing cross-site training, 
and so forth)? 

• How can the Project more effectively elicit the active support of district leadership 
(superintendents and boards of education) in Project sites? 

• How can the Project celebrate the accomplishments of teachers and administrators in its 
sites in both ceremonial and instrumental modes (i.e., for both symbolic and practical 
purposes)? 

• What can the Project do to help beginning middle-grades math and science teachers in 
Project schools (e.g., cross- or within-district subject-area mentoring, targeted in- 
service opportunities, etc.)? 

• How can the Project better respond to “more traditional” (especially late-career) 
teachers of math and science? 
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Addressing the Evaluation Question 



The strategic evaluation question asks, “What understandings of Project REAL emerge 
from the views of participants in the demonstration sites?” Detailed answers are provided in the 
stories that participants shared with the evaluator, many summarized previously, and in the 
synthesis of themes, issues, and formative questions offered in preceding sections of this report. 

So far as can be judged from the voices represented in this report, however, these 
understandings can be described as ranging from modest to deep and from somewhat to fully 
engaged. Of the approximately 25 math and science lessons observed in the case-study sites, 
perhaps 9 exhibited what the evaluator would call substantial “hands-on” content. A greater 
number (perhaps half) exhibited what the evaluator would call “rich academically-focused 
discourse.” 

In Ambler Hill, the quest for more active teaching and learning, and for a focus on 
meaning -making, seemed to the evaluator surprisingly widespread. In Ambler Valley, Project 
participants were quite clearly engaged and struggling to improve their own practice, though the 
results of efforts seemed to spread less widely, perhaps as a result of differences in organizational 
capacity (an hypothesis explored in some detail in previous discussion). 

The remaining districts (studied in less depth) seemed much more clearly to resemble 
Ambler Hill than Ambler Valley. Lower Ridges, the district from which the least amount of 
detail was gathered, showed considerable strength in the between-district comparisons from the 
cross-site teacher survey. Analysis of survey data suggests that in Lower Ridges the ideology of 
collaboration, the practice of collaboration, and project impact were strong. 

An interesting question for Project staff— and for staff of every improvement project— is 
this: “What happens when we leave?” The evaluator put this question to one of the thoughtful 
teachers interviewed. This teacher paused a bit at the question, and said it was “hard to say.” He 
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pondered a bit more and said something that made the evaluator gasp inwardly: “We probably 
would not regress because Project REAL has helped us to see we are all part of one thing.” 



Recommendations 



The evaluator believes in keeping the list of recommendations made in any evaluation 
short. Too many recommendations are worse than none, because so few can ultimately be 
addressed. The effect of numerous recommendations is to scuttle the evaluation’s key work of 
providing guidance about value or worth. 

The following recommendations are listed in order of the evaluator’s sense of their 
priority for Project sustainability. A guiding principle (or quality) other than “sustainability” 
would be predicted to change the given priority listing. All recommendations, however, are based 
on the evaluator’s experience in the demonstration sites and, more particularly, on the concerns of 
Project participants. 

1 . Sustain the level of commitment already begun. Teachers and administrators realize 
that transforming pedagogy to more “constructivist” forms is challenging work. All 
districts have begun this transformation; informants praise the Project for its 
persistence, which they contrast to the usual experience of reform (here today, gone 
tomorrow). Committed teachers and administrators, in particular, are vulnerable to 
withdrawal of Project assistance. These are excellent educators whose credibility is 
now intertwined with the commitments of the Project. These leaders require continued 
support. 

2. Improve communications on three fronts. The most critical form of commimication 
is interaction among Project participants. Means should be found to expand support for 
both formal and informal exchanges, including visits and observations among districts. 
Planners of formal training events should consider it a priority to provide time in the 
schedules of these events for informal interaction among teachers and administrators of 
the various sites. The second front concerns the communication of financial 
information and other Project administrivia from the Project headquarters to 
demonstration sites. Monthly or even quarterly statements of accounts, in addition to 
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the existing Project meetings, might suffice for this purpose. Finally, communication 
between Project headquarters and district leaders (especially superintendents) would 
probably help strengthen the hand of the Project in local sites. 

3. Assist efforts to sponsor community engagement. Rural communities are at least as 
disengaged from the academic work of schools as urban communities. Numerous paths 
to strengthening this connection exist. They include (a) making community the focus 
of curriculum; (b) launching efforts to comjnunicate with a school’s most disaffected 
parents (critical everywhere, but seldom undertaken anywhere); and (c) infusing rural 
commitments into pedagogical practice; Jamie Jackson is a local exemplar, but also 
consult the following resources for the experience of others: 

Program for Rural Services and Research (PACERS) 

The University of Alabama 

205 University Blvd. East, Box 870372, Tuscaloosa, AL 35487-0372 
voice 205-348-6432; fax 205-348-2412 

e-mail jchalmers@pacers.org (information manager; see staff page 
for other addresses); Web http://www.pacers.org/ 

PACERS works with 29 small rural public schools throughout Alabama. From the PACERS 
Web site: “Through a variety of innovative projects, PACERS schools seek to improve and change 
the nature of learning. Inherent in this program is a process of reflection and sharing between 
teachers, students, administrators, and community members that leads to genuine school reform. 
Projects involve hands-on, inter-disciplinary exercises that build on indigenous skills and 
resources. The school is recognized as the most important institution in rural communities. The 
future of rural communities is intertwined with the success of their schools." 

At this writing, the PACERS Web site details about 15 community-based projects that 
include the following, among others: aquaculture, drama, greenhouses, house construction, 
celebrations of community and place, and rural skills. The Web site also includes links to other 
programs and resources, staff and partner contacts, and contact information for all the schools 
served by PACERS (many with e-mail addresses). 

Rural School and Community Trust Place-Based Partners 
e-mail info@ruraledu.org; 

Web: http://www.ruralchallengepolicy.org/places.html 

From the Rural Trust Partner Web page: “The Rural Trust is about places — the more than 
700 schools in 33 states that are the heart of place-based education, and that are the soul of the 
rural education reform movement that is gaining momentum across the country. This section of the 
Web site will introduce you to the places where the Rural Trust and its partner organizations are 
working, and to the philosophies that guide their work. It will also allow you to contact individual 
places to learn more about their projects, which have been funded by the Rural Trust’s 
predecessor organization, the Annenberg Rural Challenge.” For additional information about the 
Rural Trust, see the Policy section of this chapter. 
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4. Gather baseline data for future evaluative purposes. This recommendation covers 

(a) possible assessment of organizational capacity (the cross-site teacher survey has 
validated a short instrument and gathered such data already; that short instrument, 
however, focuses almost exclusively on organizational capacity for collaboration) and 

(b) student-level data®® for an eventual objectives-oriented evaluation of formal project 
objectives; without data on the test performance of children in each teacher’s 
classroom, inferences about Project impact on Proficiency Test performance are 
simply not feasible. 

5. Scaffold support for improved technology use, at local option. As reported in 
study after study, training is the element most commonly neglected in hardware and 
software purchases by schools. This is a major area of need (it would not be listed 
here otherwise). Addressing this need, however, is partly dependent on efforts to 
address recommendations 1-4, above. 



Conclusion 

Interactions with participants strongly suggest that the Project is helping widely encourage 
the sorts of dispositions valued in current mathematics and science reform efforts. Data from the 
Project- wide survey of teachers show that the Project has actively engaged participants in all 
districts and, further, that Project impact is sufficiently strong to influence even teachers who do 
not teach mathematics or science. Finally, Project staff are working hard to help districts improve 
teaching and learning while simultaneously working to improve the Project’s technical assistance 
capacities and responsiveness. 




“ Desirable data would (ideally) include the following: Proficiency Test Scores for several years; 
measures of family socioeconomic status; IQ or ability scores (if available); special education status; and 
Project-related information about teachers in whose classrooms a student is instructed (e.g., view of 
hands-on-leaming. Project participation). Using such data, regression analysis is recommended as a 
workable method for detecting possible project influences. 
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Appendix A; Memorandum of Understanding 



Project REAL Evaluation: Memorandum of Understanding 

This Memorandum of Understanding has been developed to document initial assumptions 
and agreem_ent shared by the Project REAL Director and the external evaluator about the conduct 
of the evaluation of the Project. It considers the Project background and presents the evaluation 
plan. It describes the evaluator's capacity to conduct this work, estimates how evaluation 
resources will be allocated among anticipated activities, and lists deliverables and describes the 
payment schedule. 

Background 

Project REAL is a one-year project to provide impoverished districts in rural southeast 
Ohio opportunities to improve math and science instruction and student academic performance 
(for instance, on the Ohio Proficiency Tests). Among the N districts selected to receive 
assistance, five are specified for higher levels of support and are known as “demonstration sites.” 
Project activities pertain to the middle grades across all sites (grades 5-8 for the most part). 

Project REAL subscribes to contemporary thinking about instruction in math and science, 
namely, a variety of standards, curricula, and instructional and assessment methods that actively 
engage students, cultivate thinking and problem solving, and develop mathematical and scientific 
ways of thinking (in addition to factual knowledge and competence in calculation). In other 
words, the Project believes that, to some degree, changes to curriculum and instruction that 
incorporate such beliefs will improve student academic performance. 





Evaluation Plan 



The funding agency (the Ohio Department of Education) requires an evaluation of Project 
REAL, but does not specify the form such an evaluation is to take. In concert with the Project 
REAL director, the evaluator will conduct a form of “responsive evaluation” along lines 
suggested by Robert Stake (e.g., Stake, 1 975), The focus of such an evaluation is on project 
activities and the views of participants rather than on project goals or on decision making for 
management or marketing (cf. Worthen, Sanders, & Fitzpatrick, 1997). In other words, changes 
in student performance will not be assessed, nor will any judgment be made of the advisability of 
continuing the project. 

Evaluation question . The strategic evaluation question is this: “What understandings of 
Project REAL emerge from the views of participants in the demonstration sites?” This question 
enables the evaluator to develop formative judgements of the value of the Project from the 
perspective of multiple stakeholders. We expect and hope that participants will freely report a 
variety of insights about the benefits, challenges, and dilemmas of their experience. The evaluator 
and the project director both realize that school change is difficult because it entails substantial 
organizational change. 

Evaluation rationale . This evaluation approach is justified by the rationale of the program, 
which seeks to influence student performance solely through the influence of Project activities on 
the instructional capacities of staff in which participants (Project Team of the school principal and 
four teachers in each site) are located. 

Other approaches (e.g., objectives-oriented or management-oriented approaches) are 
deemed inappropriate for this project, given the fact that changes in student performance require a 
minimum of several years to be reliably observed, and more than that for changes possibly 
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attributable to intentional organizational change to materialize. 

For one-year projects, or for the initial year of multi-year projects, a responsive evaluation 
(i.e., the sort planned for Project REAL) can provide useful ^br/wat/ve judgements. That is, the 
information to be gathered in the course of this evaluation will be most useful in understanding 
the participants' experiences of Project REAL, but far less useful in understanding its outcomes. 
Project REAL staff and sponsors anticipate that project activities will improve performance of 
participating schools on the Ohio Proficiency Tests, but a single year of Project REAL activities is 
believed by us unlikely to produce this result in the short-term. The evaluator particularly 
advises all involved parties that judgement of outcomes is premature. 

Data sources . To formulate an adequate answer to the strategic question, responsive 
evaluation stresses personal interaction between participants and the evaluator. The precise 
course of such an evaluation cannot be predicted at the outset, as it is driven by concerns to be 
voiced by the participants. 

The Project Teams are to help organize professional development activities in order to 
foster schoolwide improvement in math and science instruction. For this reason, limited data 
from other teachers in the evaluation sites (the demonstration sites) will be useful in addressing 
the strategic evaluation question. 

The management of the project is also predicted to be relevant to participants' experiences 
of the project. Thus, interaction with Project staff and with their “management structures” 
(meetings and other organizational events) will require some (though not primary) attention from 
the evaluator. 

Project plans, reports, memoranda, and other documents will be relevant as well, and these 
will be made accessible to the evaluator as he may request them. Additional sources of 



documentary interest may include influential state, regional, and national resources that consider 
the improvement of instruction (e.g., professional standards, assessment practices, professional 
development). 

Evaluation activities . In order to access the data sources, the evaluator will conduct a 
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developing survey instruments (4) administering surveys, (5) conducting interviews and focus 
groups, (6) writing field notes, (7) transcribing interviews, (8) identifying issues, (9) analyzing 
data, (10) writing and submitting the final report to the Project REAL director. 

School visits will comprise the principal, but by no means exclusive, mode of accessing 
data sources. These visits may involve such events as meetings of the school-based Teams and 
professional development activities arranged by the Team, but will likely also include focus 
groups with Project Teams and informal interactions. The limited resources available for this 
evaluation, however, reduce the frequency of visits to each demonstration site to about five visits 
each. 



Confidentiality . The evaluation report will be produced by the evaluator as an internal 
document for Project RE AT. staff only. During the life of the Project, the evaluator will not 
release the report or any confidential data (data personally identifying any participant) on which 
the report is based to any other party. Upon completion and acceptance of the report, all data will 
be destroyed by the evaluator. The report will be written without personal names, though 
because contextual issues are likely to influence participants' views, it is likely that schools will 
be identifiable by a knowledgeable reader, even though the evaluator will use pseudonyms for the 
schools. 
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Allocation of Resources 



Because the precise course of the project depends on the concerns raised during the course 
of the evaluation the preceding list is just a reasonable prediction about evaluation activities. The 
evaluator will keep in mind the need to gather a variety of information and at the same time stay 
within the evaluation budget. Efficiency of data gathering is important because the contract 
budget is modest and constitutes a “fixed price.” 

Project REAL has 62.5 days— or 500 hours— of the evaluator's time at its disposal. An 
approximate allocation of this time is predicted as follows: 

Resource Allocation Table (hours) 



(1) 


making school visits 


150 (including one-way travel) 


(2) 


attending formal project activities 


50 


( 3 ) 


developing survey instruments 


20 


( 4 ) 


administering surveys 


35 


( 5 ) 


conducting interviews and focus groups 


(included in #1) 


( 6 ) 


writing field notes 


50 


( 7 ) 


transcribing interviews 


20* 


( 8 ) 


identifying issues 


35 


( 9 ) 


analyzing data 


85 


(10) 


writing and submitting the final report 


55 



* employment of temporary clerical assistance is likely here, so the time allocation listed 
here as evaluator time equivalency (i.e., 1/5 the anticipated clerical hours) 



A modest travel budget to reimburse mileage will be available, as agreed upon. The 
evaluator will charge travel time (one-way only) as project time. Return travel time will be used 
to generate audiotapes on which to base field notes. The evaluator will keep a log of time used 
and make adjustments in project activities as necessary. 

Evaluator's Capacity 

Dr. Craig Howley has conducted program evaluations for a number of organizations, 
including, most recently, the National Library of Education (“Rising Expectations,” a study of 
new directions for the national ERIC system, 1997); the University of Termessee at Martin 
(influence of teacher-developed materials on student mathematics achievement, 1994); and the 
National Science Foundation (West Virginia teachers' receptivity to telecommunications, 1993). 
Previous evaluation projects considered the effectiveness of a parental involvement program in a 
rural school district, the facility needs of a rural school district (incorporating community input), 
the suitability of different series of basal readers for learning disabled students, and alternative 
methods of identifying learning disabled college students, among others. Howley also writes 
about and conducts research on a variety of critical issues in rural education. 

Deliverables 

The evaluator will deliver an mid-course report to the Project director about March 31, 
2000. The final report is planned for submission by September 30, 2000. If variances from these 
dates are predicted, the evaluator will confer with the Project director about possible changes in 
delivery dates. The projected date for the delivery of the final report is predicated on a date of 
August 15, 2000, as a hypothetical final date for data collection. 
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Visits 



Appendix B: January 2000 Planning Document 



Observations of good and “challenged” teachers. Ambler Valley, Lower Ridges, and Ambler Hill 
will be targets. Also the Calc class in one Western Ridges high school. 

Interviews (see draft questions for protocol) 

Surveys 



1 . Cross-site teacher survey 

focus: readiness to initiate or accept change, possibly change in M/S specifically (?) may 
want to probe rural consciousness as well 

I need to survey a sample of teacher across all districts, perhaps in two groups nominated 
by the teacher leaders, one group the difficult cases and one group those thought to be at 
least very good math and science teachers. What do I ask them that might be relevant to 
the project? Do I use different instruments for each group? I'm thinking of that teacher at 
Ambler Hill that's got the “Math Lab”— how does such an innovation arise, how does or 
might the spirit behind it spill over to the rest of the school. AHSD is doing so well it 
should be a case study. Check out ETS test database & ERIC database for possible 
instruments. 

2. Survey team leaders about decision making and fmavbel project management 

Get Tom's specific input on this: What should we ask? What we be helpful to know? 
Don't do this as a survey, do as interviews? 



Cases 



1. Math Lab: Ambler Hill — agreed to do 1/14/00; SoC or similar survey as baseline 
for acceptance of innovations 



2. School Improvement: Ambler Valiev '*'* — agreed to do 1/14/00; SoC or similar 
survey as baseline for acceptance of innovations 



Appendix C: Cross-Site Teacher Survey Questionnaire 



Math and Science Improvement Project Survey 

This short questionnaire will provide information to help us evaluate the influence of a 
project that we've been part of for the past year. It's an ongoing effort to improve math and 
science performance in rural Southeast Ohio. You don't need to know anything at all about the 
project to complete the questionnaire, which should take no more than 10 or 15 minutes. 

The questionnaire does not ask for your name or your school's name. Your responses are 
completely confldential, and the overall results will be reported only in the aggregate. There are 
just 23 questions, all multiple-choice, except for today's date. 

Thanks very much for your help. 

Part I: Background Questions (6 questions) 

Today's date 

1 . Which district do you teach in? (select one) 

[] Adams County [] Northwest [] Symmes Valley [] Western [] Zane-Trace 

2. How long have you been teaching? (select one) 

[] 0-2 years [] 3-5 years [] 6-10 years 

[] 11-20 years [] 21-30 years [] longer than 30 years 

3. Which subjects do you teach? (check all that apply) 

[] science [] math [] other subjects 

4. What grade level(s) do you currently teach? (check all that apply) 

[] 4 or lower [] 5 [] 6 [] 7 [] 8 [] 9 or higher 



Math-Science Improvement Survey Questionnaire l(May 2000) 



Page 138 



5. What phrase most closely describes your opinion of “hands-on learning”? (select one) 

[] very favorable 
[] favorable 
[] lukewarm 

[] somewhat unfavorable 
[] very unfavorable 
[] not familiar with this 

6. How has your view of hands-on learning changed in the past year? (select one) 

[] much more positive 

[] somewhat more positive 

[] has not changed 

[] somewhat less positive 

[] much less positive 

[] not familiar with this, so can't say 



Part II: Project Questions (6 questions) 

Please select one answer for each of the following questions. 

1 . Are you a member of the local Project REAL team? 

[] yes [] no [] not sure 

2. How involved have you been in Project REAL activities? 

[] a lot [] some [] a little [] not at all [] don't know 

3 . Have you ever heard of Proj ect REAL? 

[] yes [] no 

4. How often have you used state or national standards (for instance, NCTM or AAAS 
materials; or Ohio Department of Education materials) to help you create lessons or 
activities for your classroom? 

[] almost daily 

[] at least weekly 

[] at least monthly 

[] less than monthly, but sometimes 

[] not at all, or not familiar with these standards 
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5. Think about changes in the way you think about teaching and learning over the past year. 
How much has Project REAL--or other recent professional development activities related 
to math and science-influenced the way you now think about teaching and learning? 

[] enormously 
[] quite a lot 

[] somewhat, but more than a little 
[] a little 
n not at all 

6. Think about your actual teaching over the past year. How much has Project REAL— or 
other recent professional development activities related to math and science-influenced 
the way you now teach? 



[] enormously 
[] quite a lot 

[] somewhat, but more than a little 
[] a little 
[] not at all 



Part III: Questions About This District (10 questions) 

For each of the following items, please indicate the single answer that best represents your 
personal reaction to the statement, based solely on your own experiences working in this school 
district. The word “educator” is used to indicate both teachers and administrators, plus other 
professionals like librarians, counselors, psychologists, and speech therapists. 



1 . Educators in this district share core beliefs about the central mission of education. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

2. Educators in this district closely agree on expectations for student learning. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

3. There is a great deal of cooperative effort among educators in this district. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

4. Educators in this district habitually share useful suggestions about curriculum, teaching 
techniques, and instructional materials. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 
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Educators in this district exhibit a clearly focused commitment to authentic curriculum 
and instruction. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

Educators in our district really focus on how well students are learning. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

I have often visited other classrooms to observe and discuss teaching and learning. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

At least one colleague regularly observes my teaching and gives me meaningful feedback 
about it. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

Educators in this district frequently discuss educational issues vvdth one another (for 
instance, educational goals, instruction, curriculum, and how students learn best). 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 

I work with at least one administrator in this district who gives me thoughtful comments 
about teaching and learning. 

[] strongly disagree [] disagree [] neutral [] agree [] strongly agree 



Thank you! 



Appendix D: Interview Protocols (2 case studies') 



Ambler Valley Interview Questions 
Teacher interview questions 

What can you tell me about Project REAL? 

Stories of change or resistance to constnictivist, hands-on methods in classroom? 
Evidence of teachers changing? 

Major obstacles for change along these lines? 

What has team recognized and what actions have you taken to change that? 

What are the intended and unintended results of REAL monthly meetings? 
[attendance has started to grow, not decline, superintendents and board members 
coming] 

What is the worst you can say about Project REAL? 

What is the best you can say about Project REAL? 

How has your teaching changed in the past two years as compared to previously? 

Did Project REAL activities influence this change in any way? How? 

What is math [or science, as appropriate] fw in your view and why is it important for kids 
to learn math [or science, as appropriate]? 

What is your frank view of the new instructional programs that are approved by 
organizations like NCTM (e.g.. Connected Math) or, in science, NSF or AAAS (e.g., 
SCIS, SEPUP, or Fast Science or FOSS)? 

All else equal, the AVSD proficiency test scores should be higher than they are, since the 
district isn't real poor. What do you frankly think is the reason for this situation? 

What do you like about teaching? What do you wish would change? 

What do you like about your own teaching? What do you wish you could change about 
your own teaching? 
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What do you think should be done to improve math and science teaching and learning in 
AVSD? 

Project REAL provided about $50K to the district for school improvement efforts, and 
asked a local team to decide how to use the money. I hope you will share your reflections 
about how the team was formed, how it operated, the decisions that were made, and/or 
the apparent results. 



Ambler Hill Interview Questions 

Teacher interview questions 

1 . What can you tell me about Project REAL? 

Stories of change or resistance to constructivist, hands-on methods in classroom? 

Evidence of teachers changing? 

Major obstacles for change along these lines? 

What has team recognized and what actions have you taken to change that? 

What are the intended and unintended results of REAL monthly meetings? [attendance has 
started to grow, not decline, superintendents and board members coming] 

2. What is the worst you can say about Project REAL? 

3. What is the best you can say about Project REAL? 

4. How has your teaching changed in the past two years as compared to previously? 

5. Did Project REAL activities influence this change in any way? How? 

6. What is math [or science, as appropriate] Iot in your view and why is it important for kids 
to leam math [or science, as appropriate]? 

7. What do you like and what do you not like about the curricula being disseminated by 
Project REAL? 

8. All else equal, the AHSD proficiency test scores should be lower than they are, since the 
district isn't affluent. What do you frankly think is the reason for this situation? 
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9. What do you like about teaching? What do you wish would change? 

10. What do you like about your own teaching? What do you wish you could change about 
your own teaching? 

1 1 . What do you think should be done to improve math and science teaching and learning in 
AHSD? 

12. Project REAL provided about $50K to the district for school improvement efforts, and 
asked a local team to decide how to use the money. I hope you will share your reflections 
about how the team was formed, how it operated, the decisions that were made, and/or the 
apparent results. 





Appendix E: Student Focus Group Protocol 



Protocol for AHSD Student Focus Group 



Lead-in 

Say something like the following: 

[introduce yourself] 

I'm here to find out about the school and the math program because the AHSD is getting some 

extra funding and the folks paying the bill have asked me to describe what Ambler 
Hill is like. Really, I just want to hear different points of view, from students, 
about what Ambler Hill is really like, as each of you see it. 

This is an important point. There are no right of wrong answers, just different stories from each 
person. Again: I'm not looking for agreement, but for the different opinions that 
different people usually have. Don't feel fimny or wrong if you have a different 
opinion from anyone else. You're a different person and you should and probably 
do all have somewhat different opinions. I really want to hear them. So please be 
as honest as you possibly can. 

I'm not revealing what any of you will say to anyone else, and I'm not even recording your names, 
so no one will ever be able to associate what you say with your name, at least not 
as a result of what I'll be writing later on. I'm going to be taking notes, by the way, 
so there won't even be a tape recording of what you say. Tomorrow I will 
summarize what I've heard today. Later on, that summary will help me write my 
report. 

Let's get started. Tell me, each of you, your first names and what you like and don't like about 
living in Ambler Hill, if anything. When we're done with the introductions, you 
can tell me what you think about the school. 



1 . What are some of the good things about this school? 

2. What are some things that you personally would like to see change about this school? 

3. What do you think about mathematics? Is it any fun ever? 

4. Has your attitude toward math ever changed? Why? 

5. What is algebra? 
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6. What are some of the good things about math instruction in this school? 

7. What are some things you'd like to see changed about math instruction in this school? 

8. Do you know more math than your parents, anyone? Why is that? 

9. When you grow up, do you want to live here or someplace else. Why? 
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Appendix F : Ambler Hill Community Member Interviews 



1 . Are you a native of this school district? 

2. Describe your own educational and occupational background. 

3. Tell me about your involvement with the Ambler Hill schools. 

4. If you went to school here, what sort of education did you get here? 

5. How is the Symmes Valley district today different and how is it the same as it was when you 
attended school here? 

6. What makes a good school, in your view? 

7. What makes a good teacher, in your view? 

8. Have you heard about Project REAL? If so, what can you tell me about it? 

9. In your view, how do its patrons regard the school system? The district leadership? The 
middle school teachers? The quality of instruction, particularly in math and science? What 
criticism do you hear of the district and do you think it's fair? 

10. Is the Ambler Hill district one community in your view? Why or why not? What would the 
center of the commimity be, if there is a center? 

11. What sorts of things does the school system need to teach children? 

12. People— or at least school teachers and administrators and professors of education— often 
contrast textbook-based instruction with project-based instruction, also sometimes called 
“hands-on learning.” Textbook-based instruction is regarded as more traditional, whereas 
hands-on learning is considered less traditional (in fact, however, the idea of learning by doing 
is very old). What do you think about these contrasting styles of teaching and learning? 

13. Have you ever discussed mathematics or science instruction with an Ambler Hill teacher or 
administrator? Can you describe what your concern was in that discussion? 

14. Is there anything else you think it would be useful for me to know about the commimity's 
devotion to, or involvement with, the district or to schooling in general? 
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Thanks for your time! 



Appendix G 



U pper Ridges, Western Ridges Interview Questions 



Teacher interview questions 

1 . What is math [or science, as appropriate] fm in your view and why is it important for kids to 
learn math [or science, as appropriate]? 

2. What do you know and think about the curricula being disseminated by Project REAL? 

3. What do you know and think about of the new instructional programs that are approved by 
organizations like NCTM (e.g.. Connected Math) or, in science, NSF or AAAS (e.g., SCIS, 
SEPUP, or Fast Science or FOSS)? 

4. What can you tell me about Project REAL? 

a. Stories of change or resistance to constructivist, hands-on methods in classroom? 

b. Evidence of teachers changing? 

c. Major obstacles for change along these lines? 

5. What do you think about school improvement (e.g., what is it? is it necessary everywhere or 
only some places? all the time or only sometimes? is it really possible and why or why not?). 

6. What is your experience of school improvement or reform projects in this school district and 
what are the impediments to school improvement in this district? 

7. What do you like about teaching? What do you wish would change? 

8. How would you describe your own teaching? What do you like about your own teaching? 
What do you wish you could change about your own teaching? 

9. What do you personally think should be done to improve math and science teaching and 
learning in WLSD? 




10. What is the worst you can say about Project REAL? 



1 1 . What is the best you can say about Project REAL? 

12. How has your teaching changed in the past two years as compared to previously? 

13. Did Project REAL activities influence this change in any way? How? 

14. Project REAL provided about $60K to the district for school improvement efforts, and asked a 
local team to decide how to use the money. I hope you will share your reflections about how 
the team was formed, how it operated, the decisions that were made, and/or the apparent 
results. 
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Appendix H: Cross-Site Questions for Project Team Site Leaders 



1 . How has Project REAL gone this year? 

2.. Have there been other math and science improvement initiatives in the district this year? 
Which ones? How would you rate Project REAL'S contribution on a scale from 1 to 10? 

3. What has the district accomplished as a result of its participation in Project REAL this year? 

4. Tell me about the teacher whose math or science teaching is most changed this year? 

5. What have you appreciated most about the Project? 

6. If you could change one thing about the Project, what would it be? 

7. What feedback for future decision making would you like me to pass on to Project staff? 

8. How would you describe the rural component of Project REAL? 

9. What would Project staff need to know about a district before selecting it (in the future) as a 
demonstration site? 

10. Is there anything else you would like to tell me about the Project or its reception in or impact 
on math and science teaching in the district? 
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Appendix I: Project Director Final Interview Protocol 



What have you appreciated most and least about directing the Project? 

W'Tiich district do you think was best prepared to benefit from the Project and why? 

What challenges have you confronted successfully and what challenges remain to be met? 
What goals will the Project address in its second year? 

If you could have changed one thing about the Project in the past year, what would it be? 

What feedback for future decision making at the state level would you most like to see 
in the final report? 

What feedback for future decision making at the university level would you most like to see 
in the final report? 



It seems to be difficult for districts to choose not to “teach to the test,” despite the fact that the 
Project and representatives of the SDE and NSF and other organizations counsel school leaders 
and teachers to “take the long view.” Do you agree? If you agree, how do you explain this 
phenomenon? If you disagree, tell me why. 



Is there anything else you would like to tell me about the Project or impact on math and science 
teaching either in the demonstration districts or at the state level! 
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